Pulogalamu ya ProHoster > Blog > Ulamuliro > Fluentd: Chifukwa chiyani ndikofunikira kukonza buffer yotulutsa

Fluentd: Chifukwa chiyani ndikofunikira kukonza buffer yotulutsa

Fluentd: Chifukwa chiyani ndikofunikira kukonza buffer yotulutsa

Masiku ano, ndizosatheka kulingalira pulojekiti yochokera Kubernetes popanda stack ya ELK, yomwe imasunga zipika za mapulogalamu onse ndi zigawo zamagulu a gululo. Pazochita zathu, timagwiritsa ntchito stack ya EFK yokhala ndi Fluentd m'malo mwa Logstash.

Fluentd ndi wosonkhanitsa zipika zamakono, wapadziko lonse lapansi yemwe akuchulukirachulukira kutchuka ndipo walowa nawo Cloud Native Computing Foundation, ndichifukwa chake vector yake yachitukuko imayang'ana kugwiritsidwa ntchito molumikizana ndi Kubernetes.

Chowonadi chogwiritsa ntchito Fluentd m'malo mwa Logstash sichisintha mawonekedwe a pulogalamuyo, komabe, Fluentd imadziwika ndi mawonekedwe ake enieni chifukwa cha kusinthasintha kwake.

Mwachitsanzo, pamene tidayamba kugwiritsa ntchito EFK mu ntchito yotanganidwa kwambiri yodula mitengo, tinayang'anizana ndi mfundo yakuti ku Kibana mauthenga ena adawonetsedwa mobwerezabwereza kangapo. M'nkhaniyi tikuuzani chifukwa chake chodabwitsachi chikuchitika komanso momwe tingathetsere vutoli.

Vuto la kubwereza zikalata

M'mapulojekiti athu, Fluentd imayikidwa ngati DaemonSet (yokhazikitsidwa yokha pagawo lililonse la gulu la Kubernetes) ndikuyang'anira zipika za stdout mu /var/log/containers. Pambuyo posonkhanitsa ndi kukonza, zipika zamtundu wa JSON zimatumizidwa ku ElasticSearch, zokwezedwa m'magulumagulu kapena mawonekedwe odziyimira pawokha, kutengera kukula kwa polojekitiyo komanso zofunikira pakugwirira ntchito komanso kulekerera zolakwika. Kibana amagwiritsidwa ntchito ngati mawonekedwe owonetsera.

Tikamagwiritsa ntchito Fluentd yokhala ndi pulogalamu yowonjezera yotulutsa, tidakumana ndi zolembedwa zina mu ElasticSearch zinali ndi zomwezo ndipo zimasiyana ndi chizindikiritso. Mutha kutsimikizira kuti uku ndikubwereza uthenga pogwiritsa ntchito chipika cha Nginx mwachitsanzo. Mu fayilo ya chipika, uthengawu ulipo mukope limodzi:

127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -

Komabe, pali zolemba zingapo mu ElasticSearch zomwe zili ndi uthengawu:

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "HgGl_nIBR8C-2_33RlQV",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "IgGm_nIBR8C-2_33e2ST",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

Komanso, pakhoza kukhala kubwereza kopitilira kawiri.

Mukukonza vutoli muzolemba za Fluentd, mutha kuwona machenjezo ambiri okhala ndi izi:

2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"

Machenjezowa amapezeka pamene ElasticSearch silingathe kuyankha pempho mkati mwa nthawi yotchulidwa ndi request_timeout parameter, chifukwa chake chidutswa cha buffer sichingachotsedwe. Zitatha izi, Fluentd amayesa kutumiza kachidutswa ka buffer ku ElasticSearch kachiwiri ndipo atayesa mosawerengeka, ntchitoyo imamaliza bwino:

2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce" 
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b" 
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"

Komabe, ElasticSearch imawona zidutswa za buffer zomwe zasamutsidwa kuti ndizopadera ndikuzipatsa _id zamitundu yosiyanasiyana panthawi yolozera. Umu ndi momwe makope a mauthenga amawonekera.

Ku Kibana zikuwoneka motere:

Fluentd: Chifukwa chiyani ndikofunikira kukonza buffer yotulutsa

Kuthetsa mavuto

Pali njira zingapo zothetsera vutoli. Chimodzi mwa izo ndi makina omwe amapangidwira mu pulogalamu yowonjezera-plugin-elasticsearch kuti apange hashi yapadera pa chikalata chilichonse. Ngati mugwiritsa ntchito makinawa, ElasticSearch izindikira kubwereza potumiza ndikuletsa zikalata zobwereza. Koma tiyenera kuganizira kuti njira yothetsera vutoli ikulimbana ndi kufufuza ndipo sikuchotsa cholakwikacho ndi kusowa kwa nthawi, choncho tinasiya kugwiritsa ntchito.

Timagwiritsa ntchito pulogalamu yowonjezera yotchinga pa Fluentd kuti tipewe kuwonongeka kwa chipika pakachitika zovuta zanthawi yayitali kapena kuchuluka kwa mitengo yodula. Ngati pazifukwa zina ElasticSearch sikutha kulemba nthawi yomweyo chikalata ku index, chikalatacho chimayikidwa pamzere ndikusungidwa pa diski. Chifukwa chake, kwa ife, kuti tichotse gwero la vuto lomwe limayambitsa cholakwika chomwe tafotokoza pamwambapa, ndikofunikira kukhazikitsa zolondola pazotsatira za buffering, pomwe buffer ya Fluentd idzakhala kukula kokwanira komanso nthawi yomweyo amakwanitsa kuyeretsedwa mu nthawi yoperekedwa.

Ndizofunikira kudziwa kuti zikhalidwe zamagawo omwe takambirana pansipa ndi amodzi pamtundu uliwonse wogwiritsa ntchito buffering pamapulagini otulutsa, chifukwa amadalira zinthu zambiri: kuzama kwa kulemba mauthenga ku chipika ndi ntchito, magwiridwe antchito a disk, network. kuchuluka kwa mayendedwe ndi bandwidth yake. Chifukwa chake, kuti mupeze zoikamo za buffer zomwe zili zoyenera pamutu uliwonse, koma osafunikira, kupewa kusaka kwakanthawi mwachimbulimbuli, mutha kugwiritsa ntchito zidziwitso zomwe Fluentd amalemba pa chipika chake pogwira ntchito ndikupeza zolondola mwachangu.

Pa nthawi yomwe vutoli linalembedwa, kasinthidwe kankawoneka motere:

 <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.test.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 8M
        queue_limit_length 8
        overflow_action block
      </buffer>

Pothetsa vutoli, mfundo za magawo otsatirawa zidasankhidwa pamanja:
chunk_limit_size - kukula kwa ma chunk omwe mauthenga mu buffer amagawidwa.

  • flush_interval - nthawi yotalikirapo pambuyo pake buffer imachotsedwa.
  • queue_limit_length - kuchuluka kwa magawo omwe ali pamzere.
  • request_timeout ndi nthawi yomwe kulumikizana pakati pa Fluentd ndi ElasticSearch kumakhazikitsidwa.

Kukula kwathunthu kwa bafa kumatha kuwerengedwa pochulukitsa magawo a queue_limit_length ndi chunk_limit_size, zomwe zitha kutanthauziridwa ngati "chiwerengero chambiri cha magawo pamzere, womwe uliwonse uli ndi kukula kwake." Ngati kukula kwa buffer sikukwanira, chenjezo lotsatirali liziwoneka m'zipika:

2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:block

Zimatanthawuza kuti buffer ilibe nthawi yoti ichotsedwe mu nthawi yomwe yaperekedwa ndipo deta yomwe imalowa mu buffer yonse imatsekedwa, zomwe zidzatsogolera kutayika kwa gawo la zipika.

Mutha kukulitsa buffer m'njira ziwiri: powonjezera kukula kwa chunk iliyonse pamzere, kapena kuchuluka kwa magawo omwe angakhale pamzere.

Mukayika chunk size chunk_limit_size kupitilira 32 megabytes, ndiye kuti ElasticSeacrh siivomereza, chifukwa paketi yomwe ikubwera idzakhala yayikulu kwambiri. Chifukwa chake, ngati mukufuna kukulitsa buffer patsogolo, ndibwino kuti muwonjezere kutalika kwa mzere wamzere_limit_length.

Chosungiracho chikasiya kusefukira ndipo uthenga wokhawokha ukakhala wosakwanira, mukhoza kuyamba kuonjezera request_timeout parameter. Komabe, ngati muyika mtengo wake ku masekondi opitilira 20, machenjezo otsatirawa ayamba kuwonekera muzolemba za Fluentd:

2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev"

Uthengawu sukhudza kagwiritsidwe ntchito ka makina mwanjira ina iliyonse ndipo zikutanthauza kuti nthawi yotulutsa buffer idatenga nthawi yayitali kuposa yomwe idakhazikitsidwa ndi slow_flush_log_threshold parameter. Izi ndikuchotsa zolakwika ndipo timazigwiritsa ntchito posankha mtengo wa request_timeout parameter.

Algorithm yosankha generalized ndi motere:

  1. Khazikitsani request_timeout pamtengo wotsimikizika kukhala wokulirapo kuposa kufunikira (mazana amasekondi). Pakukhazikitsa, mulingo waukulu pakukhazikitsa koyenera kwa parameter iyi kudzakhala kuzimiririka kwa machenjezo chifukwa chosowa nthawi.
  2. Yembekezerani mauthenga opitilira pang'onopang'ono_flush_log_threshold. Chenjezo la m'munda wa elapsed_time liwonetsa nthawi yeniyeni yomwe buffer idachotsedwa.
  3. Khazikitsani request_timeout pamtengo wokulirapo kuposa kuchuluka kwanthawi yomwe yadutsa_nthawi yomwe mwapeza panthawi yomwe mwawonera. Timawerengera mtengo wa request_timeout monga elapsed_time + 50%.
  4. Kuti muchotse machenjezo okhudza kuthamangitsidwa kwa nthawi yayitali pa chipika, mutha kukweza mtengo wa slow_flush_log_threshold. Timawerengera mtengo uwu monga kale_time + 25%.

Makhalidwe omaliza a magawo awa, monga tanenera kale, amapezedwa payekhapayekha pamilandu iliyonse. Potsatira algorithm yomwe ili pamwambayi, tatsimikiziridwa kuti tichotsa cholakwika chomwe chimatsogolera ku mauthenga obwerezabwereza.

Gome ili m'munsimu likuwonetsa momwe kuchuluka kwa zolakwika patsiku, zomwe zimatsogolera kubwereza kwa mauthenga, kusintha pakusankha zomwe zafotokozedwa pamwambapa:

node-1
node-2
node-3
node-4

Pamaso pambuyo
Pamaso pambuyo
Pamaso pambuyo
Pamaso pambuyo

analephera kutulutsa buffer
1749/2
694/2
47/0
1121/2

kuyesanso kwatheka
410/2
205/1
24/0
241/2

Ndikoyeneranso kudziwa kuti zosintha zomwe zingachitike zitha kutaya kufunikira kwake pamene polojekiti ikukula ndipo, motero, kuchuluka kwa zipika kumawonjezeka. Chizindikiro chachikulu cha kutha kwa nthawi yosakwanira ndi kubwerera kwa mauthenga okhudza kusungidwa kwa bafa kwautali ku chipika cha Fluentd, ndiko kuti, kupyola malire a slow_flush_log_threshold. Kuyambira pano, pali malire ang'onoang'ono asanayambe kupyola parameter ya request_timeout, choncho m'pofunika kuyankha mauthengawa panthawi yake ndikubwereza ndondomeko yosankha zoikidwiratu zomwe zafotokozedwa pamwambapa.

Pomaliza

Kukonza bwino buffer ya Fluentd ndi imodzi mwamagawo akulu okonzekera stack ya EFK, kutsimikizira kukhazikika kwa magwiridwe antchito ake komanso kuyika koyenera kwa zolemba m'ma index. Kutengera ndi algorithm yofotokozedwayo, mutha kutsimikiza kuti zipika zonse zidzalembedwera ku ElasticSearch index moyenerera, popanda kubwereza kapena kutayika.

Werenganinso zolemba zina pa blog yathu:

Source: www.habr.com

Kuwonjezera ndemanga