Dzina langa ndi Igor Sidorenko, ndine mtsogoleri waukadaulo mu gulu la ma admins omwe amasunga zida zonse za Domclick.

Ndikufuna kugawana zomwe ndakumana nazo pakukhazikitsa kusungidwa kwa data ku Elasticsearch. Tidzawona zomwe makonda pa node ali ndi udindo wogawa shards, momwe ILM imagwirira ntchito ndikugwira ntchito.

Omwe amagwira ntchito ndi zipika, mwanjira ina, amakumana ndi vuto la kusungirako nthawi yayitali kuti afufuze pambuyo pake. Mu Elasticsearch, izi ndi zoona makamaka, chifukwa zonse zinali zachisoni ndi magwiridwe antchito a curator. Mtundu wa 6.6 unayambitsa ntchito ya ILM. Ili ndi magawo 4:

• Hot - Mndandandawu ukusinthidwa ndikufunsidwa.
• Kutentha - Mndandandawu sunasinthidwenso, koma ukufunsidwabe.
• Zozizira - Mndandanda sunasinthidwenso ndipo safunsidwa kawirikawiri. Zambirizi ziyenera kufufuzidwabe, koma mafunso angakhale ochedwerapo.
• Chotsani - Mndandandawu sukufunikanso ndipo ukhoza kuchotsedwa.

Kupatsidwa

• Elasticsearch Data Hot: 24 processors, 128 GB memory, 1,8 TB SSD RAID 10 (8 node).
• Elasticsearch Data Warm: 24 processors, 64 GB memory, 8 TB NetApp SSD Policy (4 node).
• Elasticsearch Data Cold: 8 processors, 32 GB memory, 128 TB HDD RAID 10 (4 node).

Cholinga

Zokonda izi ndi zapayekha, zonse zimatengera malo omwe ali pa node, kuchuluka kwa ma index, mitengo, ndi zina. Tili ndi 2-3 TB ya data patsiku.

• Masiku 5 - Gawo lotentha (8 main / 1 replica).
• Masiku 20 - Gawo lofunda (kuchepa-index 4 chachikulu / 1 chojambula).
• Masiku 90 - Gawo lozizira (freeze-index 4 chachikulu / 1 chojambula).
• Masiku 120 - Chotsani gawo.

Kukhazikitsa Elasticsearch

Kuti mugawire ma shards kudutsa ma node, mumangofunika parameter imodzi yokha:

• Hot- mfundo:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: hot

• ofunda- mfundo:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: warm

• Cold- mfundo:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: cold

Kupanga Logstash

Kodi zonsezi zimagwira ntchito bwanji ndipo tinagwiritsa ntchito bwanji izi? Tiyeni tiyambe ndikupeza zipika mu Elasticsearch. Pali njira ziwiri:

1. Logstash imatenga zipika kuchokera ku Kafka. Mutha kunyamula zoyera kapena kusintha mbali yanu.
2. Chinachake chomwe chimalembera Elasticsearch, mwachitsanzo, seva ya APM.

Ganizirani chitsanzo chowongolera ma index kudzera pa Logstash. Imapanga index ndipo imagwira ntchito kwa izo index template ndi zogwirizana ILM.

k8s-ingress.conf

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # включаем управление шаблонами
        template_name => "k8s-ingress" # имя применяемого шаблона
        ilm_enabled => true # включаем управление ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
        ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
        ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
    }
}

Kupanga kwa Kibana

Pali njira yoyambira yomwe ikugwira ntchito ku ma index onse atsopano. Imayika kugawa kwa ma index otentha, kuchuluka kwa shards, replicas, etc. Kulemera kwa template kumatsimikiziridwa ndi chisankho order. Ma templates okhala ndi kulemera kwakukulu amaposa ma template omwe alipo kale kapena kuwonjezera zatsopano.

GET_template/default

{
  "default" : {
    "order" : -1, # вес шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # применяем ко всем индексам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # уровень сжатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распределяем только по горячим нодам
            },
            "total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
          }
        },
        "refresh_interval" : "5s", # интервал обновления индекса
        "number_of_shards" : "8", # количество шардов
        "auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
        "number_of_replicas" : "1" # количество реплик
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Kenako perekani mapu ku ma index k8s-ingress-* pogwiritsa ntchito template yokhala ndi kulemera kwakukulu.

Pezani _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Pambuyo pogwiritsira ntchito ma templates onse, timagwiritsira ntchito ndondomeko ya ILM ndikuyamba kuyang'anira moyo wa ma index.

GET _ilm/policy/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # теплая фаза
          "min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # куда перемещаем индекс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
            }
          }
        },
        "cold" : { # холодная фаза
          "min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # куда перемещаем индекс
              }
            },
            "freeze" : { } # замораживаем для оптимизации
          }
        },
        "hot" : { # горячая фаза
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
              "max_age" : "1d" # максимальный срок жизни индекса до ротации
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # фаза удаления
          "min_age" : "120d", # максимальный срок жизни после ротации перед удалением
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Mavuto

Panali zovuta pakukhazikitsa ndi kukonza zolakwika.

Gawo lotentha

Pakusinthasintha kolondola kwa ma indices, kupezeka kumapeto ndikofunikira index_name-date-000026 nambala zamtundu 000001. Pali mizere mu code yomwe imayang'ana ma index pogwiritsa ntchito mawu okhazikika a kukhalapo kwa manambala kumapeto. Apo ayi, padzakhala cholakwika, palibe ndondomeko zomwe zidzagwiritsidwe ntchito pa index, ndipo nthawi zonse zidzakhala mu gawo lotentha.

Gawo lofunda

Lowani (cutoff) - kuchepetsa chiwerengero cha shards, chifukwa tili ndi mfundo 4 mu magawo otentha ndi ozizira.

• Mlozera uyenera kuwerengedwa kokha.
• Kopi ya shard iliyonse mu index iyenera kukhala pa mfundo yomweyo.
• Mkhalidwe waumoyo wamagulu uyenera kukhala wobiriwira.

Kuti mudulire mlozera, Elasticsearch imasuntha ma shards onse ku mfundo imodzi, kubwereza cholozera chocheperako ndi magawo ofunikira, kenako ndikuchotsa chakale. Parameter total_shards_per_node ziyenera kukhala zofanana kapena zokulirapo kuposa kuchuluka kwa tizigawo tating'ono tokwanira pa mfundo imodzi. Apo ayi, padzakhala zidziwitso ndipo shards sizingasunthire kumalo olondola.

GET /shrink-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Gawo lozizira

amaundana (freeze) - Timayimitsa mlozera kuti tikwaniritse mafunso pazambiri zakale.

Zosaka zomwe zimachitika pazigawo zowuma zimagwiritsa ntchito ulusi waung'ono, wodzipatulira, wa search_throttled kuwongolera kuchuluka kwakusaka kwakanthawi komwe kumakhudza magawo oundana pamfundo iliyonse. Izi zimachepetsa kuchuluka kwa makumbukidwe owonjezera omwe amafunikira pama data osakhalitsa omwe amafanana ndi ma shards oundana, omwe amateteza ma node kuti asagwiritse ntchito kukumbukira kwambiri.
Ma indices owumitsidwa amawerengedwa-okha: simungathe kuloza.
Kusaka pama indices owumitsidwa kukuyembekezeka kuchitika pang'onopang'ono. Ma index owumitsidwa sanapangidwe kuti azisakasaka kwambiri. Ndizotheka kuti kusaka mlozera wozizira kumatha kutenga masekondi kapena mphindi kuti kumalize, ngakhale kusaka komweko kumalizidwa mu ma milliseconds pomwe ma indices sanawumitsidwe.

Zotsatira

Tidaphunzira momwe tingakonzekerere ma node ogwirira ntchito ndi ILM, kukhazikitsa template yogawa shards pakati pa node zotentha, ndikukhazikitsa ILM ya index ndi magawo onse a moyo.

maulalo othandiza

• https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-management-api.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/recovery-prioritization.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#indices-shrink-index
• https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#shard-allocation-awareness

Source: www.habr.com