Zita rangu ndinonzi Igor Sidorenko, ini ndiri mutungamiriri wehunyanzvi muchikwata chevatariri vanochengeta iyo yese Domklik zvivakwa mukushanda.

Ndinoda kugovera ruzivo rwangu mukugadzira yakagoverwa kuchengetedza data muElasticsearch. Tichatarisa kuti ndeapi marongero pane node ane basa rekugovera shards, kuti ILM yakagadziridzwa uye inoshanda sei.

Vaya vanoshanda nematanda ndeimwe nzira kana imwe yakatarisana nedambudziko rekuchengetedza kwenguva refu kwekuongorora kunotevera. Izvi zvinonyanya kuitika muElasticsearch nekuti kuita kwecurator kwave kusiririsa. Mushanduro 6.6, ILM kushanda kwakaonekwa. Inosanganisira 4 phases:

• Kupisa-Indekisi iri kuvandudzwa uye kubvunzwa.
• Inodziya - iyo index haisisiri kuvandudzwa, asi ichiri kubvunzwa.
• Kutonhora - Indekisi haichagadziriswe uye haiwanzo kubvunzwa. Mashoko aya anofanira kuramba achitsvakwa, asi mibvunzo inogona kunonoka.
• Delete-Indekisi haichadiwi uye inogona kubviswa zvakachengeteka.

Given

• Elasticsearch Data Inopisa: 24 processors, 128 GB yendangariro, 1,8 TB SSD RAID 10 (8 node).
• Elasticsearch Data Inodziya: 24 processors, 64 GB yendangariro, 8 TB NetApp SSD Policy (4 node).
• Elasticsearch Data Cold: 8 processors, 32 GB yendangariro, 128 TB HDD RAID 10 (4 nodes).

Chinangwa

Aya marongero ndeega, zvese zvinoenderana nenzvimbo pane node, nhamba yema indexes, matanda, nezvimwe. Kwatiri iyi 2-3 TB yedata pazuva.

• Mazuva mashanu - Kupisa chikamu (5 chikuru / 8 replica).
• Mazuva makumi maviri - Warm phase (shrink-index 4 huru / 1 replica).
• 90 mazuva - Cold phase (freeze-index 4 huru / 1 replica).
• 120 mazuva - Delete chikamu.

Kugadzira Elasticsearch

Kugovera shards pakati pemanodhi, iwe unongoda imwe parameter:

• Hot-node:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: hot

• Warm-node:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: warm

• chando-node:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: cold

Kugadzira Logstash

Izvi zvese zvinoshanda sei uye takaita sei basa iri? Ngatitange nekutora matanda muElasticsearch. Pane nzira mbiri:

1. Logstash inotora matanda kubva kuKafka. Inogona kutora yakachena kana kushandura padivi payo.
2. Chimwe chinhu chinozvinyora kune Elasticsearch, semuenzaniso, APM server.

Ngatitarisei muenzaniso wekutonga indexes kuburikidza neLogstash. Inogadzira index uye inoshanda index template uye dzinoenderana ILM.

k8s-ingress.conf

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # включаем управление шаблонами
        template_name => "k8s-ingress" # имя применяемого шаблона
        ilm_enabled => true # включаем управление ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
        ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
        ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
    }
}

Kugadzira Kibana

Iko kune yakakosha template inoshanda kune ese matsva indexes. Inoisa kugoverwa kwema indexes anopisa, nhamba ye shards, replicas, nezvimwe. Kurema kwe template kunotarirwa nechisarudzo order. Matemplate ane huremu hwepamusoro anodarika matemplate aripo kana kuwedzera matsva.

GET_template/default

{
  "default" : {
    "order" : -1, # вес шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # применяем ко всем индексам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # уровень сжатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распределяем только по горячим нодам
            },
            "total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
          }
        },
        "refresh_interval" : "5s", # интервал обновления индекса
        "number_of_shards" : "8", # количество шардов
        "auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
        "number_of_replicas" : "1" # количество реплик
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Wobva waisa mepu kune indexes k8s-ingress-* kushandisa template ine huremu hwepamusoro.

WANA _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Mushure mekushandisa ese matemplate, isu tinoshandisa iyo ILM mutemo uye tinotanga kutarisa hupenyu hweiyo indexes.

GET _ilm/policy/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # теплая фаза
          "min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # куда перемещаем индекс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
            }
          }
        },
        "cold" : { # холодная фаза
          "min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # куда перемещаем индекс
              }
            },
            "freeze" : { } # замораживаем для оптимизации
          }
        },
        "hot" : { # горячая фаза
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
              "max_age" : "1d" # максимальный срок жизни индекса до ротации
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # фаза удаления
          "min_age" : "120d", # максимальный срок жизни после ротации перед удалением
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Matambudziko

Paive nematambudziko pakuseta uye kugadzirisa nhanho.

Hot phase

Kutenderera kwakaringana kwema indexes, kuvepo pamagumo e index_name-date-000026 nhamba dzefomati 000001. Pane mitsara mukodhi inoongorora indexes uchishandisa chirevo chenguva dzose chenhamba kumagumo. Kana zvisina kudaro, pachava nekukanganisa, mitemo haizoshandisi kune index uye ichagara iri muchikamu chinopisa.

Warm phase

Shrink (kucheka) - kuderedza nhamba yemashadhi, nokuti tine nodes 4 muzvikamu zvinodziya uye zvinotonhora. Zvinyorwa zvine mitsara inotevera:

• Indekisi inofanira kuverengwa-chete.
• Kopi yega yega shard mune index inofanira kugara pane imwechete node.
• Hutano hwemasumbu hunofanirwa kunge hwakasvibira.

Kucheka index, Elasticsearch inofambisa ese mashedhi ekutanga kune imwe node, inodzokorora index yakachekwa nemaparamita anodiwa, yobva yadzima yekare. Parameter total_shards_per_node inofanira kuenzana kana kukura kudarika nhamba yemashadhi makuru kuitira kuti ikwane pane imwe nodi. Kana zvisina kudaro, pachava nezviziviso uye shards haizofambi kune node dzinodiwa.

GET /shrink-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Cold phase

-oma (freeze) - tinoomesa index kuti tigadzirise mibvunzo inoenderana nenhoroondo yedata.

Tsvagiridzo dzinoitwa pamaindikisi akaomeswa nechando dzinoshandisa diki, yakatsaurirwa, search_throttled threadpool kudzora huwandu hwetsvakiridzo dzakafanana dzinobata mashadhi akaomeswa nechando panzvimbo yega yega. Izvi zvinodzikamisa huwandu hweyekuwedzera ndangariro inodiwa kune yenguva pfupi data zvimiro zvinoenderana nechando shards, izvo zvinozodzivirira node kubva kune yakawandisa ndangariro kushandiswa.
Ma indices akaomeswa anoverengwa-chete: haugone kuisa mukati mawo.
Kutsvaga pamaindices akaomeswa nechando kunotarisirwa kuita zvishoma nezvishoma. Ma indices akaomeswa haana kuitirwa kukwidza kutsvaga. Zvinogoneka kuti tsvakiridzo yeindekisi yakaomeswa nechando inogona kutora masekonzi kana maminetsi kuti ipere, kunyangwe tsvakiridzo imwe cheteyo yakapedzwa mumamilliseconds apo indices yakanga isina kuomeswa nechando.

Migumisiro

Takadzidza kugadzirira node dzekushanda neILM, kugadzira template yekugovera shards pakati penode dzinopisa, uye gadziriso ILM ye index ine zvikamu zvese zvehupenyu.

Useful links

• https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-management-api.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/recovery-prioritization.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#indices-shrink-index
• https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#shard-allocation-awareness

Source: www.habr.com