N'oge a, ọ gaghị ekwe omume iche n'echiche ọrụ dabeere na Kubernetes na-enweghị nchịkọta ELK, nke na-echekwa ndekọ nke ngwa abụọ na usoro nke ụyọkọ ahụ. Na omume anyị, anyị na-eji EFK stack na Fluentd kama Logstash.
Fluentd bụ onye na-anakọta log ọgbara ọhụrụ, nke na-enwetawanye ewu ewu ma sonye na Cloud Native Computing Foundation, nke mere vector mmepe ya ji lekwasị anya na iji ya na Kubernetes.
Eziokwu nke iji Fluentd kama Logstash adịghị agbanwe isi ihe n'ozuzu nke ngwugwu ngwanrọ, agbanyeghị, Fluentd na-eji nuances ya akọwapụtara nke ọma sitere na ntụgharị ya.
Dị ka ihe atụ, mgbe anyị malitere iji EFK na-arụ ọrụ na-ekwo ekwo nke nwere nnukwu osisi osisi, anyị chere eziokwu ahụ ihu na na Kibana, e gosipụtara ozi ụfọdụ ugboro ugboro ugboro ugboro. N'isiokwu a, anyị ga-agwa gị ihe mere ihe a na-eme na otu esi edozi nsogbu ahụ.
Nsogbu nke oyiri akwụkwọ
N'ime ọrụ anyị, a na-ebugharị Fluentd dị ka DaemonSet (ewepụtara ya ozugbo n'otu ihe atụ na ọnụ ọnụ nke ụyọkọ Kubernetes) ma na-enyocha ndekọ akpa stdout na /var/log/containers. Mgbe nchịkọta na nhazi gasịrị, a na-eziga ndekọ n'ụdị akwụkwọ JSON na ElasticSearch, na-ebuli ya na ụyọkọ ma ọ bụ nke kwụ ọtọ, dabere na nha nke ọrụ ahụ na ihe ndị a chọrọ maka ịrụ ọrụ na ntachi obi. A na-eji Kibana dị ka interface eserese.
Mgbe ị na-eji Fluentd nwere ngwa mgbakwunye mgbakwunye mmepụta, anyị zutere ọnọdụ ebe ụfọdụ akwụkwọ dị na ElasticSearch nwere otu ọdịnaya ma dị iche na naanị na njirimara. Ị nwere ike ịchọpụta na nke a bụ ikwugharị ozi site na iji Nginx log dị ka ọmụmaatụ. N'ime faịlụ ndekọ, ozi a dị n'otu nnomi:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -Agbanyeghị, enwere ọtụtụ akwụkwọ na ElasticSearch nwere ozi a:
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "HgGl_nIBR8C-2_33RlQV",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "IgGm_nIBR8C-2_33e2ST",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}Ọzọkwa, enwere ike ịnwe ihe karịrị ugboro abụọ.
Mgbe ị na-edozi nsogbu a na ndekọ Fluentd, ị nwere ike ịhụ ọnụ ọgụgụ ịdọ aka ná ntị buru ibu na ọdịnaya ndị a:
2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"Ịdọ aka ná ntị ndị a na-eme mgbe ElasticSearch enweghị ike iweghachi nzaghachi na arịrịọ n'ime oge akọwapụtara site na paramita request_timeout, nke mere na enweghị ike ihichapụ iberibe nchekwa ebugara. Mgbe nke a gachara, Fluentd na-anwa izipu ibebe nchekwa ahụ na ElasticSearch ọzọ na mgbe ọnụọgụgụ aka ike gasịrị, ọrụ ahụ ga-emecha nke ọma:
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce"
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b"
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"Agbanyeghị, ElasticSearch na-elele ibebe nchekwa ebugharị nke ọ bụla dị ka ihe pụrụ iche wee kenye ha ụkpụrụ ubi _id pụrụ iche n'oge ndenye aha. Otu a ka akwụkwọ ozi si apụta.
Na Kibana ọ dị ka nke a:
Nchọpụta nsogbu
Enwere ọtụtụ nhọrọ iji dozie nsogbu a. Otu n'ime ha bụ usoro arụnyere n'ime ngwa mgbakwunye fluent-plugin-elasticsearch maka ịmepụta hash pụrụ iche maka akwụkwọ ọ bụla. Ọ bụrụ na ị na-eji usoro a, ElasticSearch ga-amata ugboro ugboro na ọkwa mbugharị ma gbochie akwụkwọ oyiri. Mana anyị ga-eburu n'uche na usoro a nke idozi nsogbu ahụ na-alụ ọgụ na nyocha ahụ ma ghara iwepụ njehie ahụ na enweghị oge, n'ihi ya, anyị hapụrụ iji ya.
Anyị na-eji ngwa mgbakwunye buffering na mmepụta Fluentd iji gbochie mfu log na ihe omume nke nsogbu netwọk dị mkpirikpi ma ọ bụ mmụba osisi osisi. Ọ bụrụ na n'ihi ihe ụfọdụ ElasticSearch enweghị ike ide akwụkwọ ozugbo na ndeksi, akwụkwọ ahụ na-akwụ n'ahịrị ma chekwaa ya na diski. Ya mere, n'ọnọdụ anyị, iji kpochapụ isi iyi nke nsogbu ahụ nke na-eduga na njehie ahụ akọwara n'elu, ọ dị mkpa ịtọ ụkpụrụ ziri ezi maka ihe ntinye ihe, nke Fluentd mmepụta ihe nchekwa ga-abụ nke zuru oke na nha. n'otu oge ahụ jikwaa ka ekpochapụ na oge ekenyela.
Ọ dị mma ịmara na ụkpụrụ nke paramita ndị a tụlere n'okpuru bụ nke ọ bụla n'otu n'otu n'otu n'otu nke iji buffering na plugins mmepụta, n'ihi na ha dabere n'ọtụtụ ihe: ike nke ide ozi na log site na ọrụ, arụmọrụ sistemụ diski, netwọkụ. Ibu ọwa na bandwit ya. Ya mere, iji nweta ntọala nchekwa nke dabara maka ikpe nke ọ bụla, mana ọ bụghị ejighi ya, na-ezere ogologo ọchụchọ na-enweghị isi, ị nwere ike iji ozi debugging nke Fluentd na-ede na ndekọ ya n'oge arụ ọrụ na ngwa ngwa nweta ụkpụrụ ziri ezi.
N'oge edere nsogbu ahụ, nhazi ahụ dị ka nke a:
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.test.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 8M
queue_limit_length 8
overflow_action block
</buffer>Mgbe a na-edozi nsogbu ahụ, ejiri aka họrọ ụkpụrụ nke paramita ndị a:
chunk_limit_size — nha nke chunks nke ekesara ozi dị na nchekwa.
- flush_interval - oge etiti oge emechaa kpochapụ ihe nchekwa ahụ.
- queue_limit_length — ọnụ ọgụgụ kacha nke chunks na kwụ n'ahịrị.
- request_timeout bụ oge nke emebere njikọ dị n'etiti Fluentd na ElasticSearch.
Enwere ike ịgbakọ mkpokọta ihe nchekwa site na ịba ụba queue_limit_length na chunk_limit_size, nke enwere ike ịkọwa dị ka "ọnụọgụ kacha nke chunks na kwụ n'ahịrị, nke ọ bụla nwere oke enyere." Ọ bụrụ na nha nchekwa ezughị oke, ịdọ aka ná ntị na-esonụ ga-apụta na ndekọ:
2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:blockỌ pụtara na ihe nchekwa ahụ enweghị oge ikpochapụ n'oge a na-ekenye na data nke na-abanye na nchekwa zuru ezu na-egbochi, nke ga-eduga na nkwụsị nke akụkụ nke ndekọ.
Ị nwere ike ịbawanye ihe nchekwa ahụ n'ụzọ abụọ: site n'ịbawanye ma ọ bụ nha nke ọ bụla n'ahịrị n'ahịrị, ma ọ bụ ọnụ ọgụgụ nke chunks nwere ike ịdị na kwụ n'ahịrị.
Ọ bụrụ na ịtọọ nha chunk_limit_size kariri megabytes 32, mgbe ahụ ElasticSeacrh agaghị anabata ya, ebe ngwugwu na-abata ga-adị ukwuu. Yabụ, ọ bụrụ na ịchọrọ ịbawanye ihe nchekwa ahụ n'ihu, ọ ka mma ịbawanye ogologo kwụ n'ahịrị queue_limit_length.
Mgbe ihe nchekwa ahụ kwụsịrị na-ejubiga ihe ókè na naanị ozi agwụla agwụla, ị nwere ike ịmalite ịbawanye oke arịrịọ_timeout. Agbanyeghị, ọ bụrụ na ịtọọ uru karịa 20 sekọnd, ịdọ aka ná ntị ndị a ga-amalite ịpụta na ndekọ Fluentd:
2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" Ozi a anaghị emetụta ọrụ sistemu ahụ n'ụzọ ọ bụla, ọ pụtara na oge nfefe nfefe ahụ were ogologo oge karịa ka ejiri usoro slow_flush_log_threshold setịpụrụ. Nke a bụ ozi nbipu ma anyị na-eji ya mgbe ị na-ahọrọ uru nke paramita request_timeout.
Algọridim nhọrọ mkpokọta bụ nke a:
- Tọọ request_timeout na uru ekwenyere na ọ ga-akarịrị mkpa (ọtụtụ narị sekọnd). N'oge nhazi, isi ihe maka nhazi ziri ezi nke oke a ga-abụ nkwụsị nke ịdọ aka ná ntị maka enweghị oge.
- Chere maka ozi gbasara ịgafe ọnụ ụzọ slow_flush_log_threshold. Ederede ịdọ aka ná ntị dị n'ọhịa gafere_oge ga-egosi ezigbo oge ekpochapụrụ ihe nchekwa ahụ.
- Tọọ arịrịọ_timeout ka ọ bụrụ uru karịrị oke uru oge gafere n'oge nlele ahụ. Anyị na-agbakọ uru arịrịọ_timeout dị ka oge gafere + 50%.
- Iji wepu ịdọ aka ná ntị gbasara mgbanaka nchekwa ogologo na ndekọ, ị nwere ike ibuli uru nke slow_flush_log_threshold. Anyị na-agbakọ ọnụ ahịa a ka oge gafere + 25%.
A na-enweta ụkpụrụ ikpeazụ nke paramita ndị a, dị ka e kwuru na mbụ, n'otu n'otu maka ikpe ọ bụla. Site na ịgbaso algọridim dị n'elu, a na-ekwe nkwa na anyị ga-ewepụ njehie na-eduga na ozi ugboro ugboro.
Tebụl dị n'okpuru na-egosi otú ọnụ ọgụgụ nke njehie kwa ụbọchị, na-eduga na nsụgharị nke ozi, na-agbanwe na usoro nke ịhọrọ ụkpụrụ nke paramita a kọwara n'elu:
ọnụ-1
ọnụ-2
ọnụ-3
ọnụ-4
Tupu emechaa
Tupu emechaa
Tupu emechaa
Tupu emechaa
ewepụghị ihe nchekwa ahụ
1749/2
694/2
47/0
1121/2
ịnwale ọzọ gara nke ọma
410/2
205/1
24/0
241/2
Okwesiri iburu n'uche na ntọala ndị a ga-esi na ya pụta nwere ike ịkwụsị mkpa ha ka ọrụ ahụ na-eto ma, ya mere, ọnụ ọgụgụ ndekọ na-abawanye. Isi akara nke oge ezughị oke bụ nlọghachi nke ozi gbasara ogologo nchekwa nchekwa na ndekọ Fluentd, ya bụ, gafere ọnụ ụzọ slow_flush_log_threshold. Site na nke a gaa n'ihu, a ka nwere obere oke tupu agafere paramita request_timeout, yabụ ọ dị mkpa ịzaghachi ozi ndị a n'oge kwesịrị ekwesị ma megharịa usoro nke ịhọrọ ntọala kachasị mma akọwara n'elu.
nkwubi
Ndozi nke ọma na ihe nchekwa Fluentd bụ otu n'ime usoro nhazi nke EFK stack, na-ekpebi nkwụsi ike nke ọrụ ya na ntinye akwụkwọ ziri ezi na ndenye aha. Dabere na algọridim nhazi a kọwara, ị nwere ike ijide n'aka na a ga-edebanye aha ndekọ niile na ElasticSearch index n'usoro ziri ezi, na-enweghị ikwugharị ma ọ bụ mfu.
Gụọkwa akụkọ ndị ọzọ na blọọgụ anyị:
isi: www.habr.com