Lebitso la ka ke Igor Sidorenko, ke moetapele oa botekgeniki sehlopheng sa batsamaisi ba bolokang lisebelisoa tsohle tsa Domclick.

Ke batla ho arolelana boiphihlelo ba ka ba ho theha polokelo ea data e phatlalalitsoeng ho Elasticsearch. Re tla sheba hore na ke litlhophiso life ho li-node tse ikarabellang bakeng sa ho ajoa ha li-shards, kamoo ILM e sebetsang le ho sebetsa kateng.

Ba sebetsang ka lifate, ka tsela e 'ngoe kapa e' ngoe, ba tobana le bothata ba ho boloka nako e telele bakeng sa tlhahlobo ea morao-rao. Ho Elasticsearch, sena ke 'nete haholo, hobane ntho e' ngoe le e 'ngoe e ne e le malimabe ka ts'ebetso ea curator. Mofuta oa 6.6 o hlahisitse ts'ebetso ea ILM. E na le mekhahlelo e 4:

• Ho chesa - Lenane le ntse le nchafatsoa le ho botsoa.
• Mofuthu - Lenane ha le sa ntlafatsoa, ​​empa le ntse le botsoa.
• Cold - Lenane ha le sa ntlafatsoa mme ha se hangata le botsoang. Lintlha li ntse li tlameha ho batloa, empa lipotso li kanna tsa lieha.
• Hlakola - Lenane ha le sa hlokahala 'me le ka hlakoloa ka mokhoa o sireletsehileng.

Fuoa

• Elasticsearch Data Hot: 24 processors, 128 GB memory, 1,8 TB SSD RAID 10 (8 nodes).
• Elasticsearch Data Mofuthu: 24 processors, 64 GB memori, 8 TB NetApp SSD Policy (4 nodes).
• Elasticsearch Data Cold: 8 processors, 32 GB memory, 128 TB HDD RAID 10 (4 nodes).

Sepheo

Litlhophiso tsena ke tsa motho ka mong, tsohle li ipapisitse le sebaka sa li-node, palo ea li-index, logs, joalo-joalo. Re na le 2-3 TB ea data ka letsatsi.

• Matsatsi a 5 - Mokhahlelo o chesang (8 main / 1 replica).
• Matsatsi a 20 - mohato o futhumetseng (honyela-index 4 ea sehlooho / 1 replica).
• Matsatsi a 90 - mohato o batang (lets'oao la leqhoa 4 ea sehlooho / 1 replica).
• Matsatsi a 120 - Hlakola mohato.

Ho theha Elasticsearch

Ho abela shard ho pholletsa le li-node, o hloka parameter e le 'ngoe feela:

• Hot-nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: hot

• a futhumetseng-nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: warm

• Cold-nodes:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: cold

Ho theha Logstash

Tsohle li sebetsa joang 'me re sebelisitse tšobotsi ee joang? Ha re qale ka ho kenya lits'oants'o ho Elasticsearch. Ho na le mekhoa e 'meli:

1. Logstash e lata lifate ho tsoa Kafka. E ka nka e hloekile kapa ea fetolela ka lehlakoreng la hau.
2. Ntho ka boeona e ngolla Elasticsearch, mohlala, seva sa APM.

Nahana ka mohlala oa ho laola li-index ka Logstash. E theha index mme e sebetsa ho eona mohlala oa index le tse tsamaisanang ILM.

k8s-ingress.conf

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # включаем управление шаблонами
        template_name => "k8s-ingress" # имя применяемого шаблона
        ilm_enabled => true # включаем управление ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
        ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
        ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
    }
}

Setupo sa Kibana

Ho na le mokhoa oa motheo o sebetsang ho li-index tsohle tse ncha. E beha kabo ea li-index tse chesang, palo ea li-shards, replicas, joalo-joalo. Boima ba template bo khethoa ke khetho order. Lithempleite tse nang le boima bo holimo li feta liparamente tse teng kapa li kenye tse ncha.

FUMANA _template/default

{
  "default" : {
    "order" : -1, # вес шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # применяем ко всем индексам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # уровень сжатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распределяем только по горячим нодам
            },
            "total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
          }
        },
        "refresh_interval" : "5s", # интервал обновления индекса
        "number_of_shards" : "8", # количество шардов
        "auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
        "number_of_replicas" : "1" # количество реплик
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Ebe u sebelisa 'mapa ho li-index k8s-ingress-* ho sebedisa thempleite e nang le boima bo hodimo.

FUMANA _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Kamora ho sebelisa litempele tsohle, re sebelisa leano la ILM mme re qala ho beha leihlo bophelo ba li-index.

FUMANA _ilm/policy/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # теплая фаза
          "min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # куда перемещаем индекс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
            }
          }
        },
        "cold" : { # холодная фаза
          "min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # куда перемещаем индекс
              }
            },
            "freeze" : { } # замораживаем для оптимизации
          }
        },
        "hot" : { # горячая фаза
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
              "max_age" : "1d" # максимальный срок жизни индекса до ротации
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # фаза удаления
          "min_age" : "120d", # максимальный срок жизни после ротации перед удалением
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Mathata

Ho bile le mathata sethaleng sa ho seta le ho lokisa liphoso.

Mokhahlelo o chesang

Bakeng sa ho potoloha ho nepahetseng ha li-indices, ho ba teng qetellong ho bohlokoa index_name-date-000026 linomoro tsa sebopeho 000001. Ho na le mela khoutu e hlahlobang li-index ho sebelisa polelo e tloaelehileng bakeng sa boteng ba linomoro qetellong. Ho seng joalo, ho tla ba le phoso, ha ho na maano a tla sebelisoa ho index, 'me e tla lula e le boemong bo chesang.

Mokhahlelo o futhumetseng

Fokotsa (cutoff) - ho fokotsa palo ea li-shards, hobane re na le li-node tse 4 likarolong tse futhumetseng le tse batang. Litokomane li na le mela e latelang:

• Index e tlameha ho baloa feela.
• Kopi ea shard e 'ngoe le e' ngoe ho index e tlameha ho lula sebakeng se le seng.
• Boemo ba bophelo bo botle ba lihlopha bo tlameha ho ba botala.

Ho poma index, Elasticsearch e tsamaisa lithapo tsohle tsa mantlha sebakeng se le seng, e kopiletsa index e nyenyefalitsoeng ka liparamente tse hlokahalang, ebe e hlakola ea khale. Paramethara total_shards_per_node e tlameha ho lekana kapa e kholo ho feta palo ea lithako tse kholo ho lekana node e le 'ngoe. Ho seng joalo, ho tla ba le litsebiso le li-shards li ke ke tsa fallela ho li-node tse nepahetseng.

FUMANA /shrink-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Mokhahlelo o batang

hatsele (freeze) - Re emisa index ho ntlafatsa lipotso mabapi le data ea nalane.

Lipatlisiso tse entsoeng ka li-indices tse hatselitsoeng li sebelisa "search_throttled threadpool" e nyenyane, e inehetseng ho laola palo ea lipatlisiso tse hlahang ka nako e le 'ngoe tse otlang likotoana tse hoammeng sebakeng se seng le se seng. Sena se fokotsa palo ea memori e eketsehileng e hlokehang bakeng sa lits'ebetso tsa data tsa nakoana tse tsamaellanang le li-shards tse hatselitsoeng, ka hona li sireletsa li-node khahlano le tšebeliso e feteletseng ea memori.
Li-indices tse hatselitsoeng li baloa feela: u ke ke ua li supa ho tsona.
Lipatlisiso tsa li-indices tse hatselitsoeng li lebelletsoe ho etsoa butle. Li-indices tse hatselitsoeng ha lia rereloa ho batlisisa ka bongata. Ho ka etsahala hore patlisiso ea index e hatselitsoeng e ka nka metsotsoana kapa metsotso hore e phethe, le haeba lipatlisiso tse tšoanang li phethiloe ka milliseconds ha li-indices li sa hoamisoa.

Liphello

Re ithutile ho lokisa li-node bakeng sa ho sebetsa le ILM, ho theha template bakeng sa ho aba li-shards har'a li-node tse chesang, le ho theha ILM bakeng sa index e nang le mekhahlelo eohle ea bophelo.

likhokahano tse sebetsang

• https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-management-api.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/recovery-prioritization.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#indices-shrink-index
• https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#shard-allocation-awareness

Source: www.habr.com