Matsatsing ana, ho ke ke ha khoneha ho nahana ka morero o thehiloeng ho Kubernetes ntle le stack ea ELK, e bolokang lintlha tsa likopo le likarolo tsa tsamaiso ea sehlopha. Ts'ebetsong ea rona, re sebelisa stack ea EFK e nang le Fluentd sebakeng sa Logstash.
Fluentd ke 'mokelli oa li-log oa sejoale-joale, ea lefats'e ka bophara ea ntseng a tsebahala haholo mme o kene ho Cloud Native Computing Foundation, ke ka lebaka leo vector ea eona ea nts'etsopele e tsepamisitseng maikutlo ho sebelisoa hammoho le Kubernetes.
Taba ea ho sebelisa Fluentd sebakeng sa Logstash ha e fetole moelelo o akaretsang oa sephutheloana sa software, leha ho le joalo, Fluentd e khetholloa ka menahano ea eona e ikhethang e bakoang ke ho feto-fetoha ha eona.
Ka mohlala, ha re qala ho sebelisa EFK mosebetsing o phathahaneng o nang le matla a phahameng a ho rema lifate, re ne re tobane le taba ea hore Kibana melaetsa e meng e ne e bontšoa ka makhetlo a mangata. Sehloohong sena re tla u bolella hore na ke hobane'ng ha ketsahalo ena e etsahala le mokhoa oa ho rarolla bothata.
Bothata ba ho pheta-pheta litokomane
Mererong ea rona, Fluentd e sebelisoa joalo ka DaemonSet (e hlahisitsoe ka bo eona ketsahalong e le 'ngoe sebakeng se seng le se seng sa sehlopha sa Kubernetes) mme e beha leihlo lits'oants'o tsa stdout ho /var/log/containers. Ka mor'a ho bokelloa le ho sebetsa, li-logs tsa mofuta oa litokomane tsa JSON li romelloa ho ElasticSearch, tse phahamisitsoeng ka lihlopha kapa ka mokhoa o ikemetseng, ho itšetlehile ka boholo ba morero le litlhoko tsa ts'ebetso le mamello ea liphoso. Kibana e sebelisoa e le sebopeho sa graphical.
Ha re sebelisa Fluentd e nang le sephutheloana sa buffering plugin, re kopane le boemo boo litokomane tse ling ho ElasticSearch li neng li na le litaba tse ts'oanang hantle 'me li fapana ka sekhetho feela. U ka netefatsa hore ona ke pheta-pheto ea molaetsa u sebelisa log ea Nginx e le mohlala. Ho faele ea log, molaetsa ona o teng ka kopi e le 'ngoe:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -Leha ho le joalo, ho na le litokomane tse 'maloa ho ElasticSearch tse nang le molaetsa ona:
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "HgGl_nIBR8C-2_33RlQV",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "IgGm_nIBR8C-2_33e2ST",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}Ho feta moo, ho ka ba le makhetlo a fetang a mabeli.
Ha u ntse u lokisa bothata bona ho li-log tsa Fluentd, u ka bona palo e kholo ea litemoso tse nang le litaba tse latelang:
2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"Litemoso tsena li etsahala ha ElasticSearch e sa khone ho khutlisa karabo ea kopo ka nako e boletsoeng ke request_timeout parameter, ke ka lebaka leo sekhechana sa buffer se ke keng sa hlakoloa. Kamora sena, Fluentd e leka ho romella sekhechana sa buffer ho ElasticSearch hape mme kamora liteko tse ngata, ts'ebetso e phethela ka katleho:
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce"
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b"
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"Leha ho le joalo, ElasticSearch e nka e 'ngoe le e 'ngoe ea likhechana tse fetisoang e le tse ikhethang 'me e li fa boleng bo ikhethang ba _id nakong ea indexing. Ke kamoo likopi tsa melaetsa li hlahang kateng.
Ho Kibana ho shebahala tjena:
Ho rarolla mathata
Ho na le mekhoa e mengata ea ho rarolla bothata bona. E 'ngoe ea tsona ke mochine o hahiloeng ka har'a plugin-plugin-elasticsearch plugin bakeng sa ho hlahisa hash e ikhethang bakeng sa tokomane ka' ngoe. Haeba u sebelisa mochine ona, ElasticSearch e tla lemoha ho pheta-pheta mohatong oa ho fetisa le ho thibela litokomane tse kopitsoang. Empa re tlameha ho ela hloko hore mokhoa ona oa ho rarolla bothata o loantšana le lipatlisiso mme ha o felise phoso ka ho hloka nako, kahoo re ile ra tlohela tšebeliso ea eona.
Re sebelisa plugin ea buffering ho tlhahiso ea Fluentd ho thibela tahlehelo ea log ha ho ka ba le mathata a nakoana a marang-rang kapa ho eketseha ho matla ha ho rema lifate. Haeba ka lebaka le itseng ElasticSearch e sa khone ho ngola hang-hang tokomane ho index, tokomane e emetsoe 'me e bolokiloe ho disk. Ka hona, molemong oa rona, molemong oa ho felisa mohloli oa bothata bo lebisang phosong e hlalositsoeng kaholimo, hoa hlokahala ho beha litekanyetso tse nepahetseng bakeng sa li-parameter tsa buffer, moo "Fluentd" buffer e tla ba boholo bo lekaneng. ka nako e tšoanang ho khona ho hlakisoa ka nako e behiloeng.
Ke habohlokoa ho hlokomela hore litekanyetso tsa litekanyetso tse hlalositsoeng ka tlase ke tsa motho ka mong tabeng e 'ngoe le e' ngoe ea ho sebelisa buffering ho li-plugins tse hlahisoang, kaha li itšetlehile ka lintlha tse ngata: botebo ba ho ngola melaetsa ho log ka lits'ebeletso, ts'ebetso ea disk system, marang-rang. mojaro oa mocha le bandwidth ea eona. Ka hona, bakeng sa ho fumana litlhophiso tsa buffer tse loketseng nyeoe e 'ngoe le e' ngoe, empa e sa hloke letho, ho qoba lipatlisiso tse telele ka bofofu, o ka sebelisa tlhaiso-leseling eo Fluentd e e ngollang ho log ea eona nakong ea ts'ebetso mme o fumane ka potlako boleng bo nepahetseng.
Nakong eo bothata bo neng bo ngoloa, tlhophiso e ne e shebahala tjena:
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.test.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 8M
queue_limit_length 8
overflow_action block
</buffer>Ha ho rarolla bothata, boleng ba liparamente tse latelang bo ile ba khethoa ka letsoho:
chunk_limit_size - boholo ba likarolo tseo melaetsa e ka har'a buffer e arotsoeng ho tsona.
- flush_interval - karohano ea nako kamora moo buffer e hlakotsoe.
- queue_limit_length - palo e kholo ea likarolo tse teng moleng.
- request_timeout ke nako eo khokahano lipakeng tsa Fluentd le ElasticSearch e theoang.
Kakaretso ea boholo ba buffer e ka baloa ka ho atisa liparamente queue_limit_length le chunk_limit_size, e ka hlalosoang e le "palo e ka holimo ea likotoana tse moleng, tseo e 'ngoe le e 'ngoe e nang le boholo bo fanoeng." Haeba boholo ba buffer bo sa lekana, temoso e latelang e tla hlaha ka har'a li-log:
2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:blockHo bolela hore buffer ha e na nako ea ho hlakoloa ka nako e behiloeng mme data e kenang ka botlalo e koetsoe, e leng se tla lebisa tahlehelong ea karolo ea li-log.
O ka eketsa buffer ka mekhoa e 'meli: ka ho eketsa boholo ba karolo e' ngoe le e 'ngoe moleng, kapa palo ea likarolo tse ka bang teng moleng.
Haeba u beha chunk size chunk_limit_size ho feta 32 megabytes, joale ElasticSeacrh e ke ke ea e amohela, kaha pakete e kenang e tla ba kholo haholo. Ka hona, haeba u hloka ho eketsa buffer ho ea pele, ho molemo ho eketsa bolelele ba queue_limit_length.
Ha buffer e emisa ho tlala mme ho sala molaetsa o sa lekaneng oa nako, o ka qala ho eketsa paramethara ea request_timeout. Leha ho le joalo, haeba u beha boleng ho feta metsotsoana e 20, litemoso tse latelang li tla qala ho hlaha ho li-logs tsa Fluentd:
2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" Molaetsa ona ha o ame ts'ebetso ea sistimi ka tsela efe kapa efe mme ho bolela hore nako ea ho phunya buffer e nkile nako e telele ho feta e behiloeng ke slow_flush_log_threshold parameter. Sena ke ho lokisa litaba, 'me rea e sebelisa ha re khetha boleng ba paramethara ea request_timeout.
Algorithm ea khetho e akaretsang ke e latelang:
- Beha request_timeout ho boleng bo tiisitsoeng hore bo tla ba boholo ho feta kamoo ho hlokahalang (metsotsoana e makholo). Nakong ea ho seta, ntlha e ka sehloohong ea tlhophiso e nepahetseng ea parameter ena e tla ba ho nyamela ha litemoso ka lebaka la ho hloka nako ea nako.
- Emela melaetsa e mabapi le ho feta slow_flush_log_threshold threshold. Temana ea temoso e tšimong e fetileng_nako e tla bontša nako ea sebele eo buffer e tlositsoeng ka eona.
- Beha request_timeout ho boleng bo fetang boholo ba nako e fetileng_e fumanoeng nakong eo u e shebileng. Re bala boleng ba request_timeout joalo ka nako e fetileng + 50%.
- Ho tlosa litemoso mabapi le li-buffer flushes tse telele ho log, o ka phahamisa boleng ba slow_flush_log_threshold. Re bala boleng bona joalo ka nako e fetileng + 25%.
Litekanyetso tsa ho qetela tsa liparamente tsena, joalo ka ha ho boletsoe pejana, li fumanoa ka bonngoe bakeng sa nyeoe ka 'ngoe. Ka ho latela algorithm e ka holimo, re tiisetsoa ho felisa phoso e lebisang melaetsa e pheta-phetoang.
Tafole e ka tlase e bonts'a hore na palo ea liphoso ka letsatsi, e lebisang phetisong ea melaetsa, e fetoha joang ts'ebetsong ea ho khetha boleng ba li-parameter tse hlalositsoeng ka holimo:
node-1
node-2
node-3
node-4
Pele ka mor'a
Pele ka mor'a
Pele ka mor'a
Pele ka mor'a
e hlolehile ho phunya sebafe
1749/2
694/2
47/0
1121/2
ho leka hape ho atlehile
410/2
205/1
24/0
241/2
Hape ke habohlokoa ho hlokomela hore litlhophiso tse hlahisoang li ka lahleheloa ke bohlokoa ba tsona ha morero o ntse o hōla, 'me ka lebaka leo, palo ea li-log e eketseha. Letšoao la mantlha la nako e sa lekaneng ke ho khutla ha melaetsa e mabapi le buffer flush e telele ho Fluentd log, ke hore, e fetang moeli oa slow_flush_log_threshold. Ho tloha nakong ena ho ea pele, ho ntse ho e-na le moeli o monyenyane pele parameter ea request_timeout e fetisoa, kahoo hoa hlokahala ho arabela melaetsa ena ka nako e loketseng le ho pheta mokhoa oa ho khetha litlhophiso tse nepahetseng tse hlalositsoeng ka holimo.
fihlela qeto e
Ho lokisa hantle "Fluentd output buffer" ke e 'ngoe ea mehato ea mantlha ea ho hlophisa setaki sa EFK, ho lekola botsitso ba ts'ebetso ea eona le ho beha litokomane ka nepo ho li-index. Ho ipapisitsoe le algorithm e hlalositsoeng ea tlhophiso, o ka etsa bonnete ba hore lintlha tsohle li tla ngolloa ho ElasticSearch index ka tatellano e nepahetseng, ntle le ho pheta-pheta kapa tahlehelo.
Hape bala lingoliloeng tse ling ho blog ea rona:
Source: www.habr.com