Niaj hnub no, nws tsis tuaj yeem xav txog qhov project Kubernetes uas tsis muaj ELK pawg, uas txuag cov ntaub ntawv ntawm ob daim ntawv thov thiab cov khoom siv ntawm pawg. Hauv peb qhov kev xyaum, peb siv EFK pawg nrog Fluentd tsis yog Logstash.
Fluentd yog ib tug niaj hnub, universal log collector uas tau txais ntau thiab ntau nrov thiab tau koom nrog Cloud Native Computing Foundation, uas yog vim li cas nws txoj kev loj hlob vector yog tsom rau kev siv nrog Kubernetes.
Qhov tseeb ntawm kev siv Fluentd es tsis txhob Logstash tsis hloov cov ntsiab lus dav dav ntawm cov software pob, txawm li cas los xij, Fluentd yog tus cwj pwm los ntawm nws tus kheej tshwj xeeb nuances uas tshwm sim los ntawm nws cov versatility.
Piv txwv li, thaum peb pib siv EFK nyob rau hauv ib tug tibneeg hu tauj coob project nrog ib tug siab siv ntawm kev txiav txim, peb tau ntsib nrog lub fact tias nyob rau hauv Kibana ib co lus tau tshwm sim dua ob peb zaug. Hauv tsab xov xwm no peb yuav qhia koj tias vim li cas qhov tshwm sim no tshwm sim thiab yuav daws qhov teeb meem li cas.
Qhov teeb meem ntawm cov ntaub ntawv duplication
Hauv peb cov haujlwm, Fluentd tau xa mus ua DaemonSet (automatically launched nyob rau hauv ib qho piv txwv ntawm txhua qhov ntawm Kubernetes pawg) thiab saib xyuas stdout thawv log hauv /var/log/containers. Tom qab sau thiab ua tiav, cov ntawv teev npe hauv daim ntawv ntawm JSON cov ntaub ntawv raug xa mus rau ElasticSearch, tsa hauv pawg lossis daim ntawv nyob ib leeg, nyob ntawm qhov ntsuas ntawm qhov project thiab cov kev xav tau rau kev ua haujlwm thiab kev ua txhaum cai. Kibana yog siv raws li graphical interface.
Thaum siv Fluentd nrog qhov tso zis tawm buffering plugin, peb tau ntsib qhov xwm txheej uas qee cov ntaub ntawv hauv ElasticSearch muaj cov ntsiab lus zoo ib yam thiab sib txawv tsuas yog hauv tus cim. Koj tuaj yeem paub tseeb tias qhov no yog cov lus rov hais dua siv Nginx cav ua piv txwv. Hauv cov ntaub ntawv teev tseg, cov lus no muaj nyob hauv ib daim qauv:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -Txawm li cas los xij, muaj ntau cov ntaub ntawv hauv ElasticSearch uas muaj cov lus no:
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "HgGl_nIBR8C-2_33RlQV",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "IgGm_nIBR8C-2_33e2ST",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}Tsis tas li ntawd, tuaj yeem muaj ntau tshaj li ob qhov kev rov ua dua.
Thaum kho qhov teeb meem no hauv Fluentd cav, koj tuaj yeem pom ntau cov lus ceeb toom nrog cov ntsiab lus hauv qab no:
2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"Cov lus ceeb toom no tshwm sim thaum ElasticSearch tsis tuaj yeem rov qab teb rau qhov kev thov nyob rau lub sijhawm teev tseg los ntawm qhov kev thov_timeout parameter, uas yog vim li cas qhov kev xa tawm tsis tuaj yeem raug tshem tawm. Tom qab no, Fluentd sim xa cov tsis tawg mus rau ElasticSearch dua thiab tom qab muaj ntau qhov kev sim siab, kev ua haujlwm tiav tiav:
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce"
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b"
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"Txawm li cas los xij, ElasticSearch kho txhua qhov kev hloov pauv tsis sib xws raws li qhov tshwj xeeb thiab muab lawv qhov tshwj xeeb _id teb qhov tseem ceeb thaum indexing. Qhov no yog li cas cov ntawv luam ntawm cov lus tshwm.
Hauv Kibana nws zoo li no:
Tshawb Fawb
Muaj ntau txoj hauv kev los daws qhov teeb meem no. Ib qho ntawm lawv yog cov txheej txheem tsim rau hauv qhov ua tau zoo-plugin-elasticsearch plugin rau tsim ib qho tshwj xeeb hash rau txhua daim ntawv. Yog tias koj siv cov txheej txheem no, ElasticSearch yuav lees paub qhov rov ua dua ntawm theem xa mus thiab tiv thaiv cov ntaub ntawv sib npaug. Tab sis peb yuav tsum coj mus rau hauv tus account tias txoj kev daws teeb meem no nyuaj nrog kev tshawb nrhiav thiab tsis tshem tawm qhov yuam kev nrog lub sijhawm tsis muaj sijhawm, yog li peb tso tseg nws siv.
Peb siv lub buffering plugin ntawm Fluentd tso zis los tiv thaiv cov log poob thaum muaj teeb meem hauv lub sijhawm luv luv lossis nce kev siv nkag. Yog tias vim li cas ElasticSearch tsis tuaj yeem sau ib daim ntawv mus rau qhov ntsuas tam sim, cov ntaub ntawv tau teem caij thiab khaws cia rau hauv disk. Yog li ntawd, nyob rau hauv peb cov ntaub ntawv, thiaj li yuav tshem tawm lub hauv paus ntawm qhov teeb meem uas ua rau qhov yuam kev uas tau piav saum toj no, nws yog ib qho tsim nyog yuav tsum tau teeb tsa qhov tseeb qhov tseem ceeb rau qhov tsis sib xws, qhov twg Fluentd tso zis tsis tuaj yeem muaj qhov loj txaus thiab nyob rau tib lub sij hawm tswj kom tau cleared nyob rau hauv lub sij hawm allotted.
Nws yog ib qho tsim nyog sau cia tias qhov tseem ceeb ntawm qhov tsis sib tham hauv qab no yog ib tus neeg hauv txhua qhov tshwj xeeb ntawm kev siv buffering hauv cov zis plugins, vim lawv nyob ntawm ntau yam: qhov kev siv ntawm kev sau cov lus rau lub cav los ntawm cov kev pabcuam, disk system kev ua haujlwm, network. channel load thiab nws cov bandwidth. Yog li ntawd, txhawm rau kom tau txais kev teeb tsa tsis zoo uas tsim nyog rau txhua qhov xwm txheej, tab sis tsis rov ua dua, zam kev tshawb nrhiav ntev tsis pom, koj tuaj yeem siv cov ntaub ntawv debugging uas Fluentd sau rau nws lub cav thaum lub sijhawm ua haujlwm thiab tau txais cov txiaj ntsig raug.
Thaum lub sij hawm qhov teeb meem raug kaw, lub configuration zoo li no:
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.test.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 8M
queue_limit_length 8
overflow_action block
</buffer>Thaum daws qhov teeb meem, qhov tseem ceeb ntawm cov kev txwv hauv qab no tau raug xaiv manually:
chunk_limit_size - qhov luaj li cas ntawm cov chunks uas cov lus nyob rau hauv qhov tsis yog muab faib.
- flush_interval - lub sij hawm ncua tom qab uas qhov tsis yog tshem tawm.
- queue_limit_length β qhov siab tshaj plaws ntawm cov chunks hauv kab.
- request_timeout yog lub sijhawm uas qhov kev sib txuas ntawm Fluentd thiab ElasticSearch tau tsim.
Tag nrho qhov tsis muaj qhov loj tuaj yeem suav tau los ntawm kev sib npaug ntawm cov tsis txwv queue_limit_length thiab chunk_limit_size, uas tuaj yeem txhais tau tias "qhov ntau ntawm cov chunks hauv kab, txhua qhov uas muaj qhov loj me." Yog tias qhov loj me tsis txaus, cov lus ceeb toom hauv qab no yuav tshwm sim hauv cov cav:
2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:blockNws txhais tau hais tias qhov tsis muaj sijhawm yuav raug tshem tawm hauv lub sijhawm faib thiab cov ntaub ntawv uas nkag mus rau tag nrho tsis yog raug thaiv, uas yuav ua rau poob ntawm ib feem ntawm cov cav.
Koj tuaj yeem nce qhov tsis nyob hauv ob txoj hauv kev: los ntawm kev nce qhov loj ntawm txhua chunk nyob rau hauv kab, lossis tus naj npawb ntawm chunks uas tuaj yeem nyob hauv kab.
Yog tias koj teeb tsa qhov loj me chunk_limit_size rau ntau tshaj 32 megabytes, ces ElasticSeacrh yuav tsis txais nws, vim tias cov khoom tuaj yuav loj dhau. Yog li ntawd, yog tias koj xav tau nce qhov tsis ntxiv, nws yog qhov zoo dua los ua kom qhov siab tshaj plaws kab ntev queue_limit_length.
Thaum qhov tsis txwv tsis pub dhau thiab tsuas yog lub sijhawm tsis txaus cov lus tseem tshuav, koj tuaj yeem pib nce qhov request_timeout parameter. Txawm li cas los xij, yog tias koj teeb tsa tus nqi ntau dua 20 vib nas this, cov lus ceeb toom hauv qab no yuav pib tshwm hauv Fluentd cav:
2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" Cov lus no tsis cuam tshuam rau kev ua haujlwm ntawm lub kaw lus nyob rau hauv ib txoj kev thiab txhais tau hais tias lub sij hawm buffer flush siv sij hawm ntev tshaj li teem los ntawm lub slow_flush_log_threshold parameter. Qhov no yog debugging cov ntaub ntawv thiab peb siv nws thaum xaiv tus nqi ntawm qhov request_timeout parameter.
Generalized xaiv algorithm yog raws li nram no:
- Teem kev thov_timeout rau tus nqi lav kom siab dua qhov tsim nyog (ntau pua vib nas this). Thaum lub sijhawm teeb tsa, lub ntsiab lus tseem ceeb rau qhov tseeb ntawm qhov ntsuas no yuav yog qhov ploj ntawm cov lus ceeb toom txog qhov tsis muaj sijhawm.
- Tos rau cov lus hais txog tshaj qhov slow_flush_log_threshold pib. Cov ntawv ceeb toom nyob rau hauv elapsed_time teb yuav qhia lub sijhawm tiag tiag uas tsis tau raug tshem tawm.
- Teem request_timeout rau tus nqi ntau dua qhov siab tshaj plaws elapsed_time tus nqi tau txais thaum lub sijhawm soj ntsuam. Peb xam tus nqi thov_timeout raws li elapsed_time + 50%.
- Txhawm rau tshem tawm cov lus ceeb toom txog qhov tsis sib xws ntev ntawm lub cav, koj tuaj yeem nce tus nqi ntawm slow_flush_log_threshold. Peb xam tus nqi no li elapsed_time + 25%.
Qhov kawg qhov tseem ceeb ntawm cov kev txwv no, raws li tau hais ua ntej, tau txais tus kheej rau txhua kis. Los ntawm ua raws li cov algorithm saum toj no, peb tau lees tias yuav tshem tawm qhov yuam kev uas ua rau cov lus rov qab.
Cov lus hauv qab no qhia tau hais tias tus naj npawb ntawm qhov yuam kev hauv ib hnub twg, ua rau kev sib tw ntawm cov lus, kev hloov pauv hauv cov txheej txheem ntawm kev xaiv qhov tseem ceeb ntawm cov tsis tau piav qhia saum toj no:
node-1
node-2
node-3
node-4
Ua ntej tom qab
Ua ntej tom qab
Ua ntej tom qab
Ua ntej tom qab
ua tsis tiav los yaug qhov tsis
1749/2
694/2
47/0
1121/2
rov ua tiav
410/2
205/1
24/0
241/2
Nws yog ib qho tsim nyog yuav tsum tau sau tseg tias qhov kev teeb tsa tshwm sim tuaj yeem poob lawv qhov cuam tshuam raws li qhov project loj hlob thiab, raws li, tus naj npawb ntawm cov cav nce. Lub cim tseem ceeb ntawm lub sijhawm tsis txaus yog qhov xa rov qab ntawm cov lus hais txog qhov tsis sib haum xeeb ntev mus rau Fluentd cav, uas yog, tshaj qhov pib qeeb_flush_log_threshold. Txij ntawm no mus, tseem muaj cov npoo me me ua ntej qhov kev thov_timeout tsis dhau, yog li nws yuav tsum tau teb rau cov lus no kom raws sijhawm thiab rov ua cov txheej txheem ntawm kev xaiv qhov kev pom zoo tau piav qhia saum toj no.
xaus
Fine-tuning Fluentd output buffer yog ib qho ntawm cov theem tseem ceeb ntawm kev teeb tsa EFK pawg, txiav txim siab qhov ruaj khov ntawm nws txoj haujlwm thiab qhov raug tso ntawm cov ntaub ntawv hauv indexes. Raws li tau piav qhia txog kev teeb tsa algorithm, koj tuaj yeem paub tseeb tias txhua lub cav yuav raug sau rau ElasticSearch Performance index nyob rau hauv qhov kev txiav txim kom raug, tsis muaj rov ua dua lossis poob.
Kuj nyeem lwm cov ntawv hauv peb blog:
Tau qhov twg los: www.hab.com