O loʻu igoa o Igor Sidorenko, o aʻu o se taʻitaʻi faʻapitoa i le 'au a le au pulega o loʻo tausia le atinaʻe atoa o Domclick.
Ou te manaʻo e faʻasoa loʻu poto masani i le faʻatulagaina o faʻamaumauga tuʻufaʻatasia i Elasticsearch. O le a tatou vaʻavaʻai poʻo a faʻatulagaga i nodes e nafa ma le tufatufaina o shards, pe faʻapefea ona galue ma galue le ILM.
O i latou e galulue i ogalaau, i se tasi itu poʻo se isi, e feagai ma le faʻafitauli o le teuina mo se taimi umi mo suʻesuʻega mulimuli. I le Elasticsearch, e matua moni lava lenei mea, aua o mea uma na le manuia i le galuega a le curator. Fa'aliliuga 6.6 fa'alauiloa galuega ILM. E aofia ai vaega e 4:
Vevela - O lo'o fa'afouina le fa'asinomaga ma fesiligia.
Mafanafana - E le'o toe fa'afouina le fa'asinomaga, ae o lo'o fesiligia pea.
Cold - Ua le toe fa'afouina le fa'asinomaga ma e seasea fesiligia. E tatau lava ona su'esu'e le fa'amatalaga, ae atonu e fa'agesegese ai fesili.
Aveese - Ua le toe mana'omia le fa'asinomaga ma e mafai ona tapeina ma le saogalemu.
O nei faʻatulagaga e taʻitoʻatasi, e faʻalagolago uma i le nofoaga i luga o nodes, numera o faʻamatalaga, ogalaau, ma isi. E i ai a matou 2-3 TB o faʻamaumauga i le aso.
Ina ia fa'asoa se fa'ailoga i nodes, e na'o le tasi le parakalafa e te mana'omia:
vevela-nodes:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: hot
māfanafana-nodes:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: warm
mālūlū-nodes:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: cold
Fa'atulaga Logstash
E faʻafefea ona aoga uma ma faʻapefea ona matou faʻatinoina lenei faʻaoga? Tatou amata i le mauaina o ogalaau ile Elasticsearch. E lua auala:
E aumai e Logstash ni ogalaau mai Kafka. E mafai ona piki mama pe liliu i lou itu.
O se mea lava e tusi i Elasticsearch, mo se faʻataʻitaʻiga, se APM server.
Mafaufau i se faʻataʻitaʻiga o le puleaina o faʻamatalaga e ala i Logstash. E faia ai se faasino igoa ma fa'aoga i ai mamanu faasino igoa ma fetaui ILM.
k8s-ingress.conf
input {
kafka {
bootstrap_servers => "node01, node02, node03"
topics => ["ingress-k8s"]
decorate_events => false
codec => "json"
}
}
filter {
ruby {
path => "/etc/logstash/conf.d/k8s-normalize.rb"
}
if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
grok {
match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
remove_field => "log"
}
}
else {
grok {
match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
remove_field => "log"
}
}
}
output {
elasticsearch {
id => "k8s-ingress"
hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
manage_template => true # включаем управление шаблонами
template_name => "k8s-ingress" # имя применяемого шаблона
ilm_enabled => true # включаем управление ILM
ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
}
}
Seti Kibana
O loʻo i ai se mamanu faʻavae e faʻatatau i faʻasino igoa fou uma. E faʻatulagaina le tufatufaina o faʻamatalaga vevela, numera o shards, kopi, ma isi. O le mamafa o le mamanu e fuafua i le filifiliga order. O fa'ata'ita'iga e sili atu le mamafa e fa'asili ai fa'ata'ita'iga o lo'o iai pe fa'aopoopo fo'i mea fou.
GET_template/default
{
"default" : {
"order" : -1, # вес шаблона
"version" : 1,
"index_patterns" : [
"*" # применяем ко всем индексам
],
"settings" : {
"index" : {
"codec" : "best_compression", # уровень сжатия
"routing" : {
"allocation" : {
"require" : {
"box_type" : "hot" # распределяем только по горячим нодам
},
"total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
}
},
"refresh_interval" : "5s", # интервал обновления индекса
"number_of_shards" : "8", # количество шардов
"auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
"number_of_replicas" : "1" # количество реплик
}
},
"mappings" : {
"_meta" : { },
"_source" : { },
"properties" : { }
},
"aliases" : { }
}
}
Ona faaaoga lea o le faafanua i faasino igoa k8s-ingress-* fa'aaoga se fa'ata'ita'iga e maualuga le mamafa.
A maeʻa ona faʻaogaina faʻataʻitaʻiga uma, matou te faʻaogaina le ILM policy ma amata mataʻituina le olaga o faʻamatalaga.
GET _ilm/policy/k8s-ingress
{
"k8s-ingress" : {
"version" : 14,
"modified_date" : "2020-06-11T10:27:01.448Z",
"policy" : {
"phases" : {
"warm" : { # теплая фаза
"min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"box_type" : "warm" # куда перемещаем индекс
}
},
"shrink" : {
"number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
}
}
},
"cold" : { # холодная фаза
"min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"box_type" : "cold" # куда перемещаем индекс
}
},
"freeze" : { } # замораживаем для оптимизации
}
},
"hot" : { # горячая фаза
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
"max_age" : "1d" # максимальный срок жизни индекса до ротации
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : { # фаза удаления
"min_age" : "120d", # максимальный срок жизни после ротации перед удалением
"actions" : {
"delete" : { }
}
}
}
}
}
}
Faʻafitauli
Sa i ai faʻafitauli i le seti ma le faʻapipiʻiina.
Vaega vevela
Mo le suiga sa'o o fa'ailoga, o le i ai i le pito e taua tele index_name-date-000026 numera faatulagaina 000001. O loʻo i ai laina i totonu o le code e siaki ai faʻasino igoa e faʻaaoga ai se faʻamatalaga masani mo le i ai o numera i le pito. A leai, o le ai ai se mea sese, e leai se faiga faʻavae o le a faʻaaogaina i le faasino igoa, ma o le a i ai i taimi uma i le vevela.
Vaega mafanafana
Suʻe (cutoff) - faʻaitiitia le numera o shards, ona e 4 a tatou nodes i vaega mafanafana ma malulu. O faʻamaumauga o loʻo i ai laina nei:
O le fa'ailoga e tatau ona na'o le faitau.
O se kopi o so'o se fa'ailoga i le fa'asinomaga e tatau ona nofo i luga o le node e tasi.
E tatau ona lanu meamata le tulaga o le soifua maloloina.
Ina ia teuteu se faasino igoa, Elasticsearch faʻanofoina uma vaega autu i le tasi node, faʻaluaina le faʻamaufaʻailoga faʻatasi ma mea e manaʻomia, ona tape ai lea o le mea tuai. Parameter total_shards_per_node e tatau ona tutusa pe sili atu nai lo le aofa'i o vaevaega autu e fetaui i le node e tasi. A leai, o le ai ai faʻamatalaga ma shards o le a le alu i le saʻo nodes.
GET /shrink-k8s-ingress-2020.06.06-000025/_settings
faʻaʻaisa (fa'aisa) - Matou te fa'a'aisa le fa'asinomaga e fa'amanino ai fesili i fa'amaumauga fa'asolopito.
O su'esu'ega e fa'atino i fa'ailoga fa'aaisa e fa'aoga ai le filo la'ititi, fa'apitoa, su'esu'e_throttled e pulea ai le aofa'i o su'esu'ega fa'atasi e lavea ai fasi ma'aisa i node ta'itasi. E fa'atapula'aina le aofa'i o manatuaga fa'aopoopo e mana'omia mo fa'amaumauga le tumau e fetaui ma fa'a'aisa shards, lea e puipuia ai nodes mai le fa'aogaina tele o manatua.
O fa'ailoga fa'aaisa e na'o le faitau: e le mafai ona e fa'asino i totonu.
O su'esu'ega i fa'ailoga fa'aaisa e fa'amoemoe e fa'agasolo malie. O fa'ailoga fa'aaisa e le'o fa'amoemoe mo le maualuga o le uta su'esu'e. Atonu o le su'esu'eina o se fa'ailoga fa'aaisa e ono alu ni sekone po'o ni minute e fa'amae'a ai, e tusa lava pe fa'amae'a su'esu'ega tutusa i milliseconds pe a le'i fa'aisa fa'ailoga.
O taunuʻuga
Na matou aʻoaʻoina pe faʻapefea ona saunia nodes mo le galue ma le ILM, faʻapipiʻi se faʻataʻitaʻiga mo le tufatufaina atu o fasi i totonu o nodes vevela, ma faʻapipiʻi le ILM mo se faʻasino i vaega uma o le olaga.