ProHoster > Blog > Utongi > Fluentd: nei zvakakosha kuti tine inobuda buffer

Fluentd: nei zvakakosha kuti tine inobuda buffer

Fluentd: nei zvakakosha kuti tine inobuda buffer

Mazuva ano, hazvibviri kufungidzira purojekiti yakavakirwa paKubernetes isina ELK stack, nerubatsiro rwezvipi zvinyorwa zvezvose zvinoshandiswa uye zvikamu zvehurongwa zvesumbu zvakachengetwa. Mukuita kwedu, isu tinoshandisa EFK stack neFluentd panzvimbo yeLogstash.

Fluentd inguva yemazuva ano, yakajairika-chinangwa muunganidzi iyo iri kuwedzera mukurumbira uye yakabatana neCloud Native Computing Foundation, ndosaka vector yayo yekusimudzira yakatarisana nekushandiswa pamwe chete neKubernetes.

Iyo chokwadi chekushandisa Fluentd pachinzvimbo cheLogstash haichinje iyo yakajairika essence yesoftware package, zvisinei, Fluentd ine yayo chaiyo nuances inokonzerwa nekusiyana kwayo.

Semuenzaniso, patakatanga kushandisa EFK muchirongwa chakabatikana chine kukwirira kwematanda, takanga takatarisana nenyaya yekuti muKibana mamwe mameseji anoratidzwa kakawanda. Muchikamu chino, tichakuudza kuti sei chiitiko ichi chichiitika uye nzira yekugadzirisa dambudziko racho.

Dambudziko rekudzokororwa kwegwaro

Mumapurojekiti edu, Fluentd inoiswa seDaemonSet (inongomhanya mune imwe nguva pane imwe neimwe node yeKubernetes cluster) uye inocherekedza mudziyo stdout matanda mukati /var/log/containers. Mushure mekuunganidzwa nekugadziriswa, matanda mumhando yemagwaro eJSON anotumirwa kuElasticSearch, akasimudzwa mune yakaunganidzwa kana yakamira fomu, zvichienderana nehukuru hwepurojekiti uye zvinodiwa pakuita uye kukanganisa kushivirira. Kibana inoshandiswa se graphical interface.

Kana tichishandisa Fluentd ine inobuda buffering plugin, takasangana nemamiriro ezvinhu apo mamwe magwaro muElasticSearch ane izvo zvakafanana zvemukati uye anosiyana chete muID. Iwe unogona kuve nechokwadi chekuti uku kudzokororwa kwemeseji uchishandisa muenzaniso weiyo Nginx log. Mune iyo log faira, iyi meseji iripo mune imwechete chiitiko:

127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -

Nekudaro, kune akati wandei magwaro muElasticSearch ane iyi meseji:

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "HgGl_nIBR8C-2_33RlQV",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "IgGm_nIBR8C-2_33e2ST",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

Uyezve, panogona kuva nekudzokorora kaviri.

Panguva yekugadzirisa dambudziko iri, nhamba huru yeyambiro inogona kucherechedzwa muFluentd matanda ezvinotevera zvirimo:

2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"

Yambiro idzi dzinoitika kana ElasticSearch isingakwanise kudzosera mhinduro kuchikumbiro mukati menguva yakatarwa neapplication_timeout parameter, nekuda kweiyo buffer fragment yakatumirwa haigone kucheneswa. Mushure meizvozvo, Fluentd anoedza kutumira buffer fragment kuElasticSearch zvakare, uye mushure mehuwandu hwekupokana hwekuedzazve, kuvhiya kunobudirira:

2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce" 
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b" 
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"

Nekudaro, ElasticSearch inobata chimwe nechimwe chezvimedu zvakatumirwa seyakasiyana uye inovapa yakasarudzika _id yemunda kukosha paunenge uchiita indexing. Nokudaro, makopi emashoko anooneka.

MuKibana inoita seizvi:

Fluentd: nei zvakakosha kuti tine inobuda buffer

Troubleshooting

Pane zvakawanda zvinogadzirisa dambudziko iri. Imwe yacho ndiyo dhizaini yakavakirwa muiyo fluent-plugin-elasticsearch plugin kugadzira yakasarudzika hashi yegwaro rega rega. Kana iwe ukashandisa michina iyi, ElasticSearch inoziva zvakapetwa padanho rekutumira uye kudzivirira zvinyorwa zvakapetwa. Asi mumwe haakwanisi kufuratira kuti nzira iyi yekugadzirisa dambudziko inorwisana nekuongorora uye haibvisi kukanganisa nekushayikwa kwenguva, saka takasiya kushandiswa kwayo.

Isu tinoshandisa buffering plugin pane iyo Fluentd inobuda kudzivirira kurasikirwa kwematanda muchiitiko chekudzima kwetiweki pfupi kana kuwedzera matanda. Kana, nokuda kwechimwe chikonzero, ElasticSearch haikwanise kunyora gwaro kune index, gwaro rinoiswa mumutsetse unochengetwa pa diski. Naizvozvo, kwatiri, kuti tibvise bviro yedambudziko rinotungamira mukukanganisika kwatsanangurwa pamusoro, zvinodikanwa kuseta iwo chaiwo hunhu hweiyo buffering paramita, umo iyo Fluentd inobuda buffer ichave yakakwana. saizi uye panguva imwe chete vane nguva yekujekesa panguva yakatarwa.

Izvo zvinofanirwa kucherechedzwa kuti kukosha kweiyo parameter, iyo ichakurukurwa pazasi, ndeye yega mune yega yega kesi yekushandisa buffering mune inobuda plugins, sezvo inoenderana nezvakawanda zvinhu: kusimba kwekutema mameseji nemasevhisi, kuita kwe iyo disk system, iyo network chiteshi mutoro uye bandwidth yayo. Naizvozvo, kuitira kuti uwane yakakodzera kune yega yega kesi, asi kwete yakawandisa buffer marongero, kudzivirira kutsvaga kwenguva refu kwemapofu, unogona kushandisa ruzivo rwedebug iyo Fluentd inonyora kune yayo log panguva yekushanda uye nekukasira kuwana hunhu chaihwo.

Panguva yekugadzirisa dambudziko, gadziriso yacho yakaita seizvi:

 <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.test.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 8M
        queue_limit_length 8
        overflow_action block
      </buffer>

Mukati mekugadzirisa dambudziko, hunhu hweiyo inotevera paramita dzakasarudzwa nemaoko:
chunk_limit_size - saizi yemachunks umo mameseji akatyorwa mubuffer.

  • flush_interval - nguva yenguva mushure meiyo buffer yabviswa.
  • queue_limit_length - huwandu hwehuwandu hwemachunks mumutsara.
  • request_timeout ndiyo nguva inotangwa chinongedzo pakati peFluentd neElasticSearch.

Saizi yese yebhafa inogona kuverengerwa nekuwanza queue_limit_length uye chunk_limit_size parameters, iyo inogona kududzirwa se "huwandu hwehuwandu hwemachunks mumutsara, chimwe nechimwe chine saizi yakapihwa." Kana saizi yebhafa isina kukwana, yambiro inotevera ichaonekwa mumatanda:

2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:block

Zvinoreva kuti buffer haina nguva yekujekesa munguva yakatarwa uye data inopinda mubhafa yakazara yakavharwa, izvo zvinotungamirira mukurasikirwa kwechikamu chematanda.

Unogona kuwedzera buffer nenzira mbiri: kungave nekuwedzera saizi yechunk yega yega mumutsara, kana nekuwedzera huwandu hwemachunks anogona kunge ari mumutsara.

Kana iwe ukaseta chunk_limit_size chunk size inodarika 32 megabytes, saka ElasticSeacrh haigamuchire, sezvo packet inouya ichave yakakura. Naizvozvo, kana iwe uchida kuwedzera buffer mberi, zviri nani kuwedzera kureba kweiyo queue_limit_length.

Kana iyo buffer ikamira kufashukira uye chete meseji yekushaikwa kwekupedza nguva inosara, unogona kutanga kuwedzera chikumbiro_timeout paramende. Nekudaro, kuiisa kune anopfuura 20 masekonzi kuchaita kuti inotevera yambiro kuti ioneke muFluentd logs:

2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev"

Iyi meseji haikanganise sisitimu neimwe nzira uye zvinoreva kuti nguva yekutsvaira bhafa yakatora nguva yakareba kupfuura yakatarwa neslow_flush_log_threshold parameter. Iri idebug ruzivo uye tinorishandisa kana tichitora kukosha kweiyo request_timeout parameter.

Iyo generalized sarudzo algorithm ndeiyi inotevera:

  1. Seta iyo request_timeout value kuti ivimbiswe kuve yakakura pane inodiwa (mazana emasekonzi). Panguva yekumisikidza, mucherechedzo mukuru wekurongeka kwakaringana kweiyi parameter kuchave kunyangarika kweyambiro yekushaikwa kwenguva.
  2. Mirira mameseji pamusoro pekudarika slow_flush_log_threshold threshold. Muzvinyorwa zveyambiro, iyo elapsed_time field ichange iine chaiyo buffer clearing time.
  3. Seta kukosha_kwenguva yekupedza kuti ive yakakura kudarika iyo yakapfuura_nguva kukosha yakagamuchirwa panguva yekutarisisa. Isu tinoverenga iyo request_timeout value seyakapfuura_nguva + 50%.
  4. Kuti ubvise yambiro nezve buffer refu kubva parogi, unogona kusimudza iyo slow_flush_log_threshold value. Isu tinoverenga kukosha uku sekupfuura_nguva + 25%.

Iwo ekupedzisira kukosha kweaya ma paramita, sezvambotaurwa, anowanikwa ega pane imwe neimwe kesi. Kutevera algorithm iri pamusoro, isu tinovimbiswa kubvisa chikanganiso chinotungamira kune anodzokororwa mameseji.

Tafura iri pazasi inoratidza kuti huwandu hwezvikanganiso pazuva, zvinotungamira kune duplicate mameseji, shanduko mukuita kwekusarudza kukosha kweiyo paramita inotsanangurwa pamusoro:

node-1
node-2
node-3
node-4

Pamberi mushure
Pamberi mushure
Pamberi mushure
Pamberi mushure

akatadza kuburitsa buffer
1749/2
694/2
47/0
1121/2

kuedzazve kwakabudirira
410/2
205/1
24/0
241/2

Inofanirawo kucherechedzwa kuti zvigadziriso zvakagamuchirwa zvinogona kurasikirwa nekukosha kwavo mukukura kweprojekiti uye, maererano, kuwedzera kwenhamba yematanda. Mucherechedzo wekutanga wekuti nguva yekupedza haisi kurongwa kudzoserwa kwemashoko akareba e-buffer kuFluentd log, i.e. slow_flush_log_threshold threshold yadarika. Kubva panguva ino zvichienda mberi, kuchine chikamu chidiki chisati chapfuura chikumbiro_timeout parameter, saka zvinodikanwa kuti upindure kune aya mameseji panguva yakakodzera uye kudzokorora maitiro ekusarudza akanakisa marongero anotsanangurwa pamusoro.

mhedziso

Kunyatsogadzirisa iyo Fluentd inobuda buffer nderimwe rematanho makuru mukugadzirisa EFK stack, kuona kugadzikana kwekushanda kwayo uye nekuiswa kwayo kwemagwaro muma indexes. Tichitarisa pane yakatsanangurwa gadziriso algorithm, unogona kuve nechokwadi chekuti matanda ese anonyorwa kune ElasticSearch index mune iyo chaiyo kurongeka, pasina kudzokorora uye kurasikirwa.

Verengawo zvimwe zvinyorwa pane yedu blog:

Source: www.habr.com

Voeg