Kuv lub npe yog Igor Sidorenko, Kuv yog tus thawj coj hauv pab pawg tswj hwm uas tswj hwm tag nrho cov txheej txheem ntawm Domclick.

Kuv xav qhia kuv qhov kev paub dhau los hauv kev teeb tsa cov ntaub ntawv xa tawm hauv Elasticsearch. Peb yuav saib seb qhov chaw ntawm cov nodes yog lub luag haujlwm rau kev faib cov shards, ILM ua haujlwm thiab ua haujlwm li cas.

Cov neeg uas ua haujlwm nrog cov cav, ib txoj kev lossis lwm qhov, ntsib teeb meem ntawm kev khaws cia ntev rau kev txheeb xyuas tom qab. Hauv Elasticsearch, qhov no yog qhov tseeb tshwj xeeb, vim tias txhua yam tsis muaj hmoo nrog tus curator functionality. Version 6.6 qhia ILM ua haujlwm. Nws muaj 4 theem:

• Kub - Qhov ntsuas tau raug kho tshiab thiab nug.
• Sov - Qhov Performance index tsis hloov tshiab, tab sis tseem tab tom nug.
• Txias - Qhov Performance index tsis hloov tshiab thiab tsis tshua muaj lus nug. Cov ntaub ntawv tseem yuav tsum tau tshawb nrhiav, tab sis cov lus nug yuav qeeb dua.
• Rho tawm - Qhov ntsuas tsis xav tau ntxiv lawm thiab tuaj yeem raug tshem tawm kom nyab xeeb.

Muab

• Elasticsearch Cov Ntaub Ntawv Kub: 24 processors, 128 GB nco, 1,8 TB SSD RAID 10 (8 nodes).
• Elasticsearch Cov ntaub ntawv sov: 24 processors, 64 GB nco, 8 TB NetApp SSD Txoj Cai (4 nodes).
• Elasticsearch Cov Ntaub Ntawv Txias: 8 processors, 32 GB nco, 128 TB HDD RAID 10 (4 nodes).

Lub hom phiaj

Cov kev teeb tsa no yog tus kheej, nws tag nrho nyob ntawm qhov chaw ntawm cov nodes, tus naj npawb ntawm qhov ntsuas, cav, thiab lwm yam. Peb muaj 2-3 TB ntawm cov ntaub ntawv ib hnub twg.

• 5 hnub - Kub theem (8 lub ntsiab / 1 replica).
• 20 hnub - sov theem (shrink-index 4 lub ntsiab / 1 replica).
• 90 hnub - Mob khaub thuas (khov-index 4 lub ntsiab / 1 replica).
• 120 hnub - Tshem tawm theem.

Teeb tsa Elasticsearch

Txhawm rau faib cov shard hla cov nodes, koj tsuas yog xav tau ib qho parameter xwb:

• kub- nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: hot

• sov- nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: warm

• Khaub thuas- nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: cold

Teeb tsa Logstash

Nws ua haujlwm li cas thiab peb ua li cas thiaj li siv tau qhov no? Cia peb pib los ntawm kev nkag mus rau hauv Elasticsearch. Muaj ob txoj hauv kev:

1. Logstash nqa cav los ntawm Kafka. tuaj yeem khaws huv lossis hloov pauv ntawm koj sab.
2. Ib yam dab tsi nws tus kheej sau rau Elasticsearch, piv txwv li, APM server.

Xav txog ib qho piv txwv ntawm kev tswj cov indexes los ntawm Logstash. Nws tsim ib qho index thiab siv rau nws index qauv thiab sib raug ILM.

k8s-ingress.conf ib

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # включаем управление шаблонами
        template_name => "k8s-ingress" # имя применяемого шаблона
        ilm_enabled => true # включаем управление ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
        ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
        ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
    }
}

Kev teeb tsa

Muaj cov qauv hauv paus uas siv rau txhua qhov ntsuas tshiab. Nws teev cov kev faib tawm ntawm cov ntsuas kub, tus naj npawb ntawm shards, replicas, thiab lwm yam. Qhov hnyav ntawm tus qauv yog txiav txim los ntawm kev xaiv order. Cov qauv uas muaj qhov hnyav dua hla cov qauv uas twb muaj lawm lossis ntxiv cov tshiab.

GET_template/default

{
  "default" : {
    "order" : -1, # вес шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # применяем ко всем индексам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # уровень сжатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распределяем только по горячим нодам
            },
            "total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
          }
        },
        "refresh_interval" : "5s", # интервал обновления индекса
        "number_of_shards" : "8", # количество шардов
        "auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
        "number_of_replicas" : "1" # количество реплик
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Tom qab ntawd siv daim ntawv qhia rau cov indexes k8s-ingress-* siv tus qauv nrog qhov hnyav dua.

GET _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Tom qab siv tag nrho cov qauv, peb siv txoj cai ILM thiab pib saib xyuas lub neej ntawm cov indexes.

GET _ilm/policy/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # теплая фаза
          "min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # куда перемещаем индекс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
            }
          }
        },
        "cold" : { # холодная фаза
          "min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # куда перемещаем индекс
              }
            },
            "freeze" : { } # замораживаем для оптимизации
          }
        },
        "hot" : { # горячая фаза
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
              "max_age" : "1d" # максимальный срок жизни индекса до ротации
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # фаза удаления
          "min_age" : "120d", # максимальный срок жизни после ротации перед удалением
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Teeb meem

Muaj teeb meem ntawm lub teeb thiab debugging theem.

Kub theem

Rau qhov tseeb kev sib hloov ntawm indices, lub xub ntiag ntawm qhov kawg yog qhov tseem ceeb index_name-date-000026 format tooj 000001. Muaj cov kab hauv cov cai uas txheeb xyuas cov indexes siv cov lus qhia tsis tu ncua rau qhov muaj tus lej kawg. Txwv tsis pub, yuav muaj qhov yuam kev, tsis muaj cov cai yuav raug siv rau qhov ntsuas, thiab nws yuav ib txwm nyob rau theem kub.

Kub theem

Ntsws (cutoff) — txo cov shards, vim peb muaj 4 nodes nyob rau hauv lub sov thiab txias theem. Cov ntaub ntawv muaj cov kab hauv qab no:

• Qhov ntsuas yuav tsum tau nyeem nkaus xwb.
• Ib daim qauv ntawm txhua qhov shard hauv qhov ntsuas yuav tsum nyob ntawm tib lub node.
• Pawg kev noj qab haus huv yuav tsum yog ntsuab.

Txhawm rau txiav qhov Performance index, Elasticsearch txav tag nrho cov thawj shards mus rau ib lub node, duplicates lub truncated index nrog rau qhov tsim nyog tsis, thiab ces deletes cov qub. Parameter total_shards_per_node yuav tsum yog sib npaug los yog ntau tshaj tus naj npawb ntawm cov shards tseem ceeb kom haum rau ntawm ib qho. Txwv tsis pub, yuav muaj kev ceeb toom thiab shards yuav tsis txav mus rau qhov tseeb nodes.

Tau txais /shrink-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Lub caij txias

Nkoog (khov) - Peb khov qhov ntsuas kom ua kom zoo dua cov lus nug ntawm cov ntaub ntawv keeb kwm.

Kev tshawb nrhiav tau ua nyob rau hauv qhov ntsuas khov siv qhov me me, mob siab rau, search_throttled threadpool los tswj tus naj npawb ntawm cov kev tshawb fawb sib txuam uas ntaus khov shards ntawm txhua qhov ntawm. Qhov no txwv tus nqi ntawm lub cim xeeb ntxiv uas yuav tsum tau muaj rau cov ntaub ntawv hloov pauv hloov pauv mus rau cov khoom khov khov, uas yog li tiv thaiv cov nodes tiv thaiv kev nco ntau dhau.
Frozen indices yog nyeem nkaus xwb: koj tsis tuaj yeem ntsuas rau lawv.
Kev tshawb fawb ntawm cov ntsuas khov yuav tsum ua kom qeeb. Frozen indices tsis yog npaj rau kev tshawb nrhiav siab. Nws yog qhov ua tau tias kev tshawb fawb ntawm qhov ntsuas khov yuav siv sijhawm vib nas this lossis feeb kom tiav, txawm tias tib qhov kev tshawb nrhiav ua tiav hauv milliseconds thaum cov ntsuas tsis khov.

Cov txiaj ntsim tau los

Peb kawm yuav ua li cas npaj cov nodes rau kev ua hauj lwm nrog ILM, teeb ib tug template rau faib shards ntawm kub nodes, thiab teeb tsa ILM rau ib tug Performance index nrog tag nrho cov theem ntawm lub neej.

Pab kev sib txuas lus

• https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-management-api.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/recovery-prioritization.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#indices-shrink-index
• https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#shard-allocation-awareness

Tau qhov twg los: www.hab.com