Fluentd: Aisea e taua ai le fetuutuunai o le faʻapipiʻi faʻapipiʻi

I aso nei, e le mafai ona mafaufauina se poloketi faʻavae Kubernetes e aunoa ma le ELK stack, lea e faʻasaoina ai ogalaau o talosaga uma ma vaega o le vaega. I la matou faʻataʻitaʻiga, matou te faʻaogaina le EFK stack ma Fluentd nai lo Logstash.

O le Fluentd ose fa'aonaponei fa'aputu ogalaau fa'aonaponei o lo'o fa'ateleina ma sili atu ona ta'uta'ua ma ua auai i le Cloud Native Computing Foundation, o le mea lea e taula'i atu ai le fa'aoga fa'atasi ma Kubernetes.

O le mea moni o le faʻaaogaina o le Fluentd nai lo le Logstash e le suia ai le uiga lautele o le pusa polokalama, ae ui i lea, o le Fluentd e faʻaalia i ona lava nuances faʻapitoa e mafua mai i lona agavaʻa.

Mo se faʻataʻitaʻiga, ina ua amata ona matou faʻaogaina le EFK i se galuega pisi ma le maualuga o le logging, sa matou feagai ma le mea moni e faapea i Kibana o nisi o feʻau na faʻaalia pea i le tele o taimi. I lenei tusiga o le a matou taʻu atu ia te oe pe aisea e tupu ai lenei mea ma pe faapefea ona foia le faafitauli.

Le fa'afitauli o le fa'aluaina o pepa

I a matou poloketi, o loʻo faʻapipiʻiina le Fluentd e avea o se DaemonSet (e faʻaalia otometi i le tasi faʻataʻitaʻiga i pona taʻitasi o le kulupu Kubernetes) ma mataʻituina faʻamaumauga o pusa i /var/log/containers. A maeʻa le aoina ma le faʻaogaina, o ogalaau i foliga o pepa JSON e auina atu i le ElasticSearch, faʻatupuina i le fuifui poʻo le tuʻufua, e faʻatatau i le fua o le poloketi ma manaʻoga mo le faʻatinoga ma le faʻapalepale sese. O Kibana o loʻo faʻaaogaina e fai ma ata faʻataʻitaʻi.

I le fa'aogaina o le Fluentd ma se mea fa'apipi'i fa'apipi'i, na matou fa'afeiloa'i i se tulaga e iai fa'amaumauga i le ElasticSearch e tutusa lelei lava le anotusi ma e na'o le fa'ailoga e ese ai. E mafai ona e faʻamaonia o se feʻau toe fai e faʻaaoga ai le Nginx log e fai ma faʻataʻitaʻiga. I le faila ogalaau, o loʻo i ai lenei feʻau i se kopi e tasi:

127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -

Ae ui i lea, e tele pepa i le ElasticSearch o loʻo i ai lenei feʻau:

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "HgGl_nIBR8C-2_33RlQV",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "IgGm_nIBR8C-2_33e2ST",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

E le gata i lea, e mafai ona sili atu ma le lua toe fai.

A'o fa'aleleia lenei fa'afitauli i totonu o fa'amaumauga a le Fluentd, e mafai ona e va'ai i le tele o fa'asalalauga fa'atasi ai ma mea nei:

2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"

O nei lapataiga e tupu pe a le mafai e le ElasticSearch ona toe faafoi se tali i se talosaga i totonu o le taimi o loʻo faʻamaonia e le request_timeout parameter, ma o le mea lea e le mafai ai ona faʻamalo le vaega o le pa puipui na tuʻuina atu. A maeʻa lenei mea, e taumafai le Fluentd e toe lafo le vaega faʻapipiʻi i le ElasticSearch ma a maeʻa le numera o taumafaiga, e maeʻa manuia le taʻaloga:

2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce" 
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b" 
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"

Ae ui i lea, e faʻaogaina e ElasticSearch vaega taʻitasi o paʻu faʻafeiloaʻi e tulaga ese ma tuʻuina atu ia i latou tulaga faʻapitoa _id fanua i le taimi o le faasino igoa. O le ala lea e aliali mai ai kopi o fe'au.

I Kibana e pei o lenei:

Fofo faafitauli

E tele filifiliga e foia ai lenei faafitauli. O se tasi o ia mea o le masini ua fausia i totonu o le fluent-plugin-elasticsearch plugin mo le fausiaina o se hash tulaga ese mo pepa taitasi. Afai e te faʻaogaina lenei masini, ElasticSearch o le a faʻaalia le toe faia i le taimi e tuʻuina atu ai ma puipuia pepa faʻalua. Ae e tatau ona tatou manatua o lenei auala e foia ai le faafitauli e tauivi ma le suʻesuʻega ma e le faʻaumatia le mea sese ma le leai o se taimi, o lea na matou lafoaia ai lona faʻaaogaina.

Matou te fa'aogaina se fa'apipi'i fa'apipi'i i luga o le galuega a le Fluentd e puipuia ai le gau o ogalaau pe a tula'i mai fa'afitauli feso'ota'iga mo taimi pupuu po'o le fa'atupula'ia o le fa'aputuina o la'au. Afai o nisi mafuaaga e le mafai ai e ElasticSearch ona vave tusia se pepa i le faasino igoa, o le pepa o loʻo tuʻuina ma teuina i luga o le disk. O le mea lea, i la matou tulaga, ina ia faʻaumatia le faʻapogai o le faʻafitauli lea e oʻo atu ai i le mea sese o loʻo faʻamatalaina i luga, e manaʻomia le setiina o tau saʻo mo faʻamaufaʻailoga, lea o le a lava le tele o le Fluentd output buffer ma i le taimi lava e tasi pulea e kilia i le taimi atofaina.

E taua le maitauina o tau o faʻamaufaʻailoga o loʻo talanoaina i lalo e taʻitoʻatasi i tulaga taʻitasi taʻitasi o le faʻaogaina o le faʻapipiʻiina i masini faʻapipiʻi, aua latou te faʻalagolago i le tele o mea: o le malosi o le tusiaina o feʻau i le ogalaau e auaunaga, faʻatinoga o le disk, fesoʻotaiga. uta alavai ma lona bandwidth. O le mea lea, ina ia mafai ona maua faʻamaufaʻailoga e fetaui lelei mo mataupu taʻitasi, ae le o le faʻaaogaina, aloese mai suʻesuʻega uumi ma le tauaso, e mafai ona e faʻaogaina faʻamatalaga faʻamatalaga e tusia e Fluentd i lana ogalaau i le taimi o le taotoga ma vave maua le tau saʻo.

I le taimi na faʻamauina ai le faʻafitauli, o le faʻatulagaga e pei o lenei:

 <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.test.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 8M
        queue_limit_length 8
        overflow_action block
      </buffer>

Pe a foia le faʻafitauli, o tau o faʻamaufaʻailoga nei na filifilia ma le lima:
chunk_limit_size - o le tele o pusi e vaevaeina ai feʻau o loʻo i totonu o le pa.

• flush_interval - taimi vaeluaga pe a uma ona kilia le pa.
• queue_limit_length - o le numera aupito maualuga o pusi i le laina.
• request_timeout o le taimi lea e faʻamautu ai le fesoʻotaʻiga i le va o Fluentd ma ElasticSearch.

E mafai ona fa'atatau le aofa'i o le pa'u e ala i le fa'ateleina o fa'amaufa'ailoga queue_limit_length ma le chunk_limit_size, lea e mafai ona fa'auigaina o le "le numera aupito maualuga o pusi i totonu o le laina, e tofu lava ma le lapo'a." Afai e le lava le tele o le pa, o le lapataiga lea o le a aliali mai i totonu o ogalaau:

2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:block

O lona uiga e leai se taimi e faʻamama ai le paʻu i le taimi atofaina ma o faʻamaumauga e ulufale i totonu o le paʻu atoa ua poloka, lea o le a mafua ai le leiloa o se vaega o ogalaau.

E mafai ona e fa'ateleina le pa'u i ni auala se lua: e ala i le fa'ateleina o le tele o pusi ta'itasi i le laina, po'o le numera o pusi e mafai ona i ai i le laina.

Afai e te setiina le chunk size chunk_limit_size i le sili atu i le 32 megabytes, ona le talia lea e ElasticSeacrh, talu ai o le a tele naua le pepa o loʻo sau. O le mea lea, afai e te manaʻomia le faʻateleina o le paʻu, e sili atu le faʻateleina o le laina maualuga queue_limit_length.

A uma le fa'agata o le pa'u ma na'o le taimi e le lava le fe'au o lo'o totoe, e mafai ona e amata fa'ateleina le request_timeout parameter. Ae peita'i, afai e te setiina le tau i le sili atu i le 20 sekone, o lapata'iga nei o le a amata ona aliali mai i totonu o fa'amaumauga Fluentd:

2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" 

O lenei fe'au e le a'afia ai le fa'agaioia o le faiga i so'o se auala ma o lona uiga na umi atu le taimi e fa'afefeteina ai le taimi na fa'atulaga e le slow_flush_log_threshold parakalafa. Ole fa'amatalaga lea e fa'apipi'i ma matou fa'aogaina pe a filifilia le tau o le request_timeout parameter.

Ole algorithm filifiliga lautele e fa'apea:

1. Seti le request_time out i se tau e mautinoa e sili atu nai lo le mana'omia (seselau o sekone). I le taimi o le seti, o le faʻataʻitaʻiga autu mo le faʻatulagaina saʻo o lenei parakalafa o le a mou atu o lapataiga mo le leai o se taimi malolo.
2. Fa'atali mo fe'au e uiga i le sili atu i le slow_flush_log_threshold threshold. O le fa'amatalaga lapatai i le elapsed_time field o le a fa'aalia ai le taimi tonu na kilia ai le pa.
3. Seti le request_time out i se tau e sili atu nai lo le maualuga o le elapsed_time tau na maua i le vaitaimi o le mata'ituina. Matou te fa'atatau le tau ole talosaga_taimi e fa'apea elapsed_time + 50%.
4. Ina ia aveese faʻamatalaga e uiga i faʻamalo umi paʻu mai le ogalaau, e mafai ona e siitia le tau ole slow_flush_log_threshold. Matou te fa'atatauina lenei tau o le elapsed_time + 25%.

O tau fa'ai'uga o nei ta'iala, e pei ona ta'ua muamua, e maua ta'ito'atasi mo mataupu ta'itasi. E ala i le mulimuli i le algorithm o loʻo i luga, ua faʻamaonia i matou e faʻaumatia le mea sese e taʻitaʻia ai feʻau fai soo.

O le laulau o loʻo i lalo o loʻo faʻaalia ai le numera o mea sese i le aso, e oʻo atu ai i le faʻaluaina o feʻau, suiga i le faagasologa o le filifilia o tau o faʻasologa o loʻo faʻamatalaina i luga:

node-1
node-2
node-3
node-4

A'o le'i mae'a
A'o le'i mae'a
A'o le'i mae'a
A'o le'i mae'a

ua le mafai ona fufulu le pa
1749/2
694/2
47/0
1121/2

toe taumafai manuia
410/2
205/1
24/0
241/2

E taua foi le matauina o le taunuuga o tulaga e ono leiloa lo latou taua ao tuputupu ae le poloketi ma, e tusa ai, o le numera o ogalaau e faateleina. O le fa'ailoga muamua o le le lava o le taimi fa'agata o le toe fa'afo'i mai lea o fe'au e uiga i le umi o le pa'u fa'afefe i le Fluentd log, o lona uiga, sili atu i le slow_flush_log_threshold threshold. Mai lea taimi i luga, o loʻo i ai pea se laʻititi laʻititi aʻo leʻi sili atu le faʻasologa o le request_timeout, o lea e tatau ai ona tali atu i nei feʻau i se taimi talafeagai ma toe fai le faagasologa o le filifilia o tulaga sili ona lelei o loʻo faʻamatalaina i luga.

iʻuga

O le fa'amalieina o le Fluentd output buffer o se tasi lea o la'asaga autu o le fa'atulagaina o le fa'aputuga o le EFK, e fuafua ai le mautu o lona fa'agaioiga ma le tu'uina sa'o o pepa i fa'asinomaga. Faʻavae i luga o le faʻamatalaga faʻatulagaina algorithm, e mafai ona e mautinoa o le a tusia uma ogalaau i le ElasticSearch index i le faʻasologa saʻo, e aunoa ma le toe faia poʻo le leiloa.

Faitau foi isi tala i la matou blog:

• Alu ma Zabbix 5.0 na avea ma uo
• Fa'aleleia se fuifui Kubernetes e aunoa ma se taimi malolo
• Kubernetes: aisea e taua tele ai le faʻatulagaina o le puleaina o punaoa?
• Togafiti Faigofie e Tolu e Fa'aitiitia Ata o Docker
• Lagolagoina o se numera tele o poloketi 'upega tafaʻilagi eseese

puna: www.habr.com