Orukọ mi ni Igor Sidorenko, Mo jẹ oludari imọ-ẹrọ ninu ẹgbẹ ti awọn admins ti o ṣetọju gbogbo awọn amayederun ti Domclick.
Mo fẹ lati pin iriri mi ni ṣiṣeto ibi ipamọ data pinpin ni Elasticsearch. A yoo wo iru awọn eto lori awọn apa jẹ lodidi fun pinpin awọn shards, bawo ni ILM ṣe n ṣiṣẹ ati ṣiṣẹ.
Awọn ti n ṣiṣẹ pẹlu awọn akọọlẹ, ọna kan tabi omiiran, koju iṣoro ti ipamọ igba pipẹ fun itupalẹ nigbamii. Ni Elasticsearch, eyi jẹ otitọ paapaa, nitori pe ohun gbogbo jẹ lailoriire pẹlu iṣẹ ṣiṣe olutọju. Ẹya 6.6 ṣafihan iṣẹ ṣiṣe ILM. O ni awọn ipele mẹrin:
- Gbona - Atọka naa ti ni imudojuiwọn ni itara ati beere.
- Gbona - Atọka ko ni imudojuiwọn mọ, ṣugbọn o tun n beere lọwọ rẹ.
- Tutu - Atọka ko ni imudojuiwọn mọ ati pe o ṣọwọn beere. Alaye naa gbọdọ tun jẹ wiwa, ṣugbọn awọn ibeere le lọra.
- Paarẹ - Atọka naa ko nilo ati pe o le paarẹ lailewu.
Fun
- Elasticsearch Data Gbona: 24 nse, 128 GB iranti, 1,8 TB SSD RAID 10 (8 apa).
- Elasticsearch Data Gbona: 24 nse, 64 GB iranti, 8 TB NetApp SSD Afihan (4 apa).
- Elasticsearch Data Tutu: 8 nse, 32 GB iranti, 128 TB HDD RAID 10 (4 apa).
Ero
Awọn eto wọnyi jẹ ẹni kọọkan, gbogbo rẹ da lori aaye lori awọn apa, nọmba awọn atọka, awọn akọọlẹ, ati bẹbẹ lọ. A ni 2-3 TB ti data fun ọjọ kan.
- 5 ọjọ - Hot alakoso (8 akọkọ / 1 ajọra).
- Awọn ọjọ 20 - ipele ti o gbona ( 4 akọkọ / 1 ajọra).
- Awọn ọjọ 90 - ipele tutu ( 4 akọkọ / 1 ajọra).
- 120 ọjọ - Pa alakoso.
Ṣiṣeto Elasticsearch
Lati pin pinpin kaakiri awọn apa, o nilo paramita kan nikan:
- Hot- awọn apa:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr # Add custom attributes to the node: node.attr.box_type: hot - gbona- awọn apa:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr # Add custom attributes to the node: node.attr.box_type: warm - tutu- awọn apa:
~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr # Add custom attributes to the node: node.attr.box_type: cold
Ṣiṣeto Logstash
Bawo ni gbogbo rẹ ṣe ṣiṣẹ ati bawo ni a ṣe ṣe imuse ẹya yii? Jẹ ki a bẹrẹ nipa gbigba awọn iforukọsilẹ sinu Elasticsearch. Awọn ọna meji lo wa:
- Logstash fetches àkọọlẹ lati Kafka. Le gbe mimọ tabi yipada ni ẹgbẹ rẹ.
- Nkankan funrararẹ kọwe si Elasticsearch, fun apẹẹrẹ, olupin APM kan.
Wo apẹẹrẹ ti iṣakoso awọn atọka nipasẹ Logstash. O ṣẹda atọka ati ki o kan si o ati ibaramu .
k8s-ingress.conf
input {
kafka {
bootstrap_servers => "node01, node02, node03"
topics => ["ingress-k8s"]
decorate_events => false
codec => "json"
}
}
filter {
ruby {
path => "/etc/logstash/conf.d/k8s-normalize.rb"
}
if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
grok {
match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
remove_field => "log"
}
}
else {
grok {
match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
remove_field => "log"
}
}
}
output {
elasticsearch {
id => "k8s-ingress"
hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
manage_template => true # включаем управление шаблонами
template_name => "k8s-ingress" # имя применяемого шаблона
ilm_enabled => true # включаем управление ILM
ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
}
}Kibana iṣeto ni
Ilana ipilẹ wa ti o kan gbogbo awọn atọka tuntun. O ṣeto pinpin awọn atọka gbigbona, nọmba awọn shards, awọn ẹda, ati bẹbẹ lọ. Iwọn ti awoṣe jẹ ipinnu nipasẹ aṣayan order. Awọn awoṣe ti o ni iwuwo ti o ga ju awọn paramita awoṣe ti o wa tẹlẹ tabi ṣafikun awọn tuntun.
GET _awoṣe/aiyipada
{
"default" : {
"order" : -1, # вес шаблона
"version" : 1,
"index_patterns" : [
"*" # применяем ко всем индексам
],
"settings" : {
"index" : {
"codec" : "best_compression", # уровень сжатия
"routing" : {
"allocation" : {
"require" : {
"box_type" : "hot" # распределяем только по горячим нодам
},
"total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
}
},
"refresh_interval" : "5s", # интервал обновления индекса
"number_of_shards" : "8", # количество шардов
"auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
"number_of_replicas" : "1" # количество реплик
}
},
"mappings" : {
"_meta" : { },
"_source" : { },
"properties" : { }
},
"aliases" : { }
}
}Lẹhinna lo aworan agbaye si awọn atọka k8s-ingress-* lilo awoṣe pẹlu iwuwo ti o ga julọ.
GET _awoṣe/k8s-ingress
{
"k8s-ingress" : {
"order" : 100,
"index_patterns" : [
"k8s-ingress-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "k8s-ingress",
"rollover_alias" : "k8s-ingress"
},
"codec" : "best_compression",
"routing" : {
"allocation" : {
"require" : {
"box_type" : "hot"
}
}
},
"number_of_shards" : "8",
"number_of_replicas" : "1"
}
},
"mappings" : {
"numeric_detection" : false,
"_meta" : { },
"_source" : { },
"dynamic_templates" : [
{
"all_fields" : {
"mapping" : {
"index" : false,
"type" : "text"
},
"match" : "*"
}
}
],
"date_detection" : false,
"properties" : {
"kubernetes" : {
"type" : "object",
"properties" : {
"container_name" : {
"type" : "keyword"
},
"container_hash" : {
"index" : false,
"type" : "keyword"
},
"host" : {
"type" : "keyword"
},
"annotations" : {
"type" : "object",
"properties" : {
"value" : {
"index" : false,
"type" : "text"
},
"key" : {
"index" : false,
"type" : "keyword"
}
}
},
"docker_id" : {
"index" : false,
"type" : "keyword"
},
"pod_id" : {
"type" : "keyword"
},
"labels" : {
"type" : "object",
"properties" : {
"value" : {
"type" : "keyword"
},
"key" : {
"type" : "keyword"
}
}
},
"namespace_name" : {
"type" : "keyword"
},
"pod_name" : {
"type" : "keyword"
}
}
},
"@timestamp" : {
"type" : "date"
},
"nginx" : {
"type" : "object",
"properties" : {
"access" : {
"type" : "object",
"properties" : {
"agent" : {
"type" : "text"
},
"response_code" : {
"type" : "integer"
},
"upstream" : {
"type" : "object",
"properties" : {
"port" : {
"type" : "keyword"
},
"name" : {
"type" : "keyword"
},
"response_lenght" : {
"type" : "integer"
},
"response_time" : {
"index" : false,
"type" : "text"
},
"addr" : {
"type" : "keyword"
},
"status" : {
"index" : false,
"type" : "text"
}
}
},
"method" : {
"type" : "keyword"
},
"http_version" : {
"type" : "keyword"
},
"bytes_sent" : {
"type" : "integer"
},
"request_lenght" : {
"type" : "integer"
},
"url" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
},
"remote_user" : {
"type" : "text"
},
"referrer" : {
"type" : "text"
},
"remote_ip" : {
"type" : "ip"
},
"request_time" : {
"format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
"type" : "date"
},
"host" : {
"type" : "keyword"
},
"time" : {
"format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
"type" : "date"
}
}
},
"error" : {
"type" : "object",
"properties" : {
"server" : {
"type" : "keyword"
},
"upstream" : {
"type" : "object",
"properties" : {
"port" : {
"type" : "keyword"
},
"proto" : {
"type" : "keyword"
},
"host" : {
"type" : "keyword"
},
"url" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
}
}
},
"method" : {
"type" : "keyword"
},
"level" : {
"type" : "keyword"
},
"http_version" : {
"type" : "keyword"
},
"pid" : {
"index" : false,
"type" : "integer"
},
"message" : {
"type" : "text"
},
"tid" : {
"index" : false,
"type" : "keyword"
},
"url" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
},
"referrer" : {
"type" : "text"
},
"remote_ip" : {
"type" : "ip"
},
"connection_id" : {
"index" : false,
"type" : "keyword"
},
"host" : {
"type" : "keyword"
},
"time" : {
"format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
"type" : "date"
}
}
}
}
},
"log" : {
"type" : "text"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"eventtime" : {
"type" : "float"
}
}
},
"aliases" : { }
}
}Lẹhin lilo gbogbo awọn awoṣe, a lo eto imulo ILM ati bẹrẹ mimojuto igbesi aye awọn atọka.
GET _ilm / imulo / k8s-ingress
{
"k8s-ingress" : {
"version" : 14,
"modified_date" : "2020-06-11T10:27:01.448Z",
"policy" : {
"phases" : {
"warm" : { # теплая фаза
"min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"box_type" : "warm" # куда перемещаем индекс
}
},
"shrink" : {
"number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
}
}
},
"cold" : { # холодная фаза
"min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"box_type" : "cold" # куда перемещаем индекс
}
},
"freeze" : { } # замораживаем для оптимизации
}
},
"hot" : { # горячая фаза
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
"max_age" : "1d" # максимальный срок жизни индекса до ротации
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : { # фаза удаления
"min_age" : "120d", # максимальный срок жизни после ротации перед удалением
"actions" : {
"delete" : { }
}
}
}
}
}
}Isoro
Awọn iṣoro wa ni iṣeto ati ipele ti n ṣatunṣe aṣiṣe.
Hot alakoso
Fun yiyi to tọ ti awọn atọka, wiwa ni ipari jẹ pataki index_name-date-000026 awọn nọmba kika 000001. Awọn ila wa ninu koodu ti o ṣayẹwo awọn atọka nipa lilo ikosile deede fun wiwa awọn nọmba ni ipari. Bibẹẹkọ, aṣiṣe yoo wa, ko si awọn eto imulo ti yoo lo si atọka, ati pe yoo wa nigbagbogbo ni ipele ti o gbona.
Ipele gbona
Mimu (cutoff) - idinku nọmba awọn shards, nitori a ni awọn apa 4 ni awọn ipele ti o gbona ati tutu. Iwe naa ni awọn ila wọnyi:
- Atọka gbọdọ jẹ kika-nikan.
- Ẹda ti gbogbo shard ninu atọka gbọdọ gbe lori ipade kanna.
- Ipo ilera iṣupọ gbọdọ jẹ alawọ ewe.
Lati ge atọka kan, Elasticsearch gbe gbogbo awọn shards akọkọ lọ si oju ipade kan, ṣe ẹda atọka ti ge pẹlu awọn aye pataki, lẹhinna paarẹ eyi atijọ. Paramita total_shards_per_node gbọdọ jẹ dogba si tabi tobi ju nọmba awọn shards akọkọ lati baamu lori ipade kan. Bibẹẹkọ, awọn iwifunni yoo wa ati awọn shards kii yoo gbe si awọn apa to tọ.
GET / isunki-k8s-ingress-2020.06.06-000025/_eto
{
"shrink-k8s-ingress-2020.06.06-000025" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"auto_expand_replicas" : "0-1",
"blocks" : {
"write" : "true"
},
"provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
"creation_date" : "1592225525569",
"priority" : "100",
"number_of_replicas" : "1",
"uuid" : "psF4MiFGQRmi8EstYUQS4w",
"version" : {
"created" : "7060299",
"upgraded" : "7060299"
},
"lifecycle" : {
"name" : "k8s-ingress",
"rollover_alias" : "k8s-ingress",
"indexing_complete" : "true"
},
"codec" : "best_compression",
"routing" : {
"allocation" : {
"initial_recovery" : {
"_id" : "_Le0Ww96RZ-o76bEPAWWag"
},
"require" : {
"_id" : null,
"box_type" : "cold"
},
"total_shards_per_node" : "8"
}
},
"number_of_shards" : "4",
"routing_partition_size" : "1",
"resize" : {
"source" : {
"name" : "k8s-ingress-2020.06.06-000025",
"uuid" : "gNhYixO6Skqi54lBjg5bpQ"
}
}
}
}
}
}Igba otutu
di (di) - A di atọka lati mu awọn ibeere pọ si lori data itan.
Awọn awọrọojulówo ti a ṣe lori awọn atọka tio tutunini lo kekere, igbẹhin, threadpool search_throttled lati ṣakoso nọmba awọn wiwa nigbakanna ti o kọlu awọn ọpa tio tutunini lori ipade kọọkan. Eyi ṣe opin iye afikun iranti ti o nilo fun awọn ẹya data igba diẹ ti o baamu si awọn shards tio tutunini, eyiti o ṣe aabo awọn apa lodi si agbara iranti ti o pọ ju.
Awọn atọka tio tutuni jẹ kika-nikan: o ko le ṣe atọka sinu wọn.
Awọn wiwa lori awọn atọka tutunini ni a nireti lati ṣiṣẹ laiyara. Awọn atọka tutunini kii ṣe ipinnu fun fifuye wiwa giga. O ṣee ṣe pe wiwa atọka tio tutunini le gba iṣẹju-aaya tabi iṣẹju lati pari, paapaa ti awọn iwadii kanna ba pari ni milliseconds nigbati awọn atọka naa ko di didi.
Awọn esi
A kọ ẹkọ bi a ṣe le mura awọn apa fun ṣiṣẹ pẹlu ILM, ṣeto awoṣe fun pinpin awọn shards laarin awọn apa gbigbona, ati ṣeto ILM fun atọka pẹlu gbogbo awọn ipele ti igbesi aye.
wulo awọn ọna asopọ
orisun: www.habr.com