Jina langu ni Igor Sidorenko, mimi ni kiongozi wa kiufundi katika timu ya wasimamizi ambao wanadumisha miundombinu yote ya Domclick.

Ninataka kushiriki uzoefu wangu katika kusanidi hifadhi ya data iliyosambazwa katika Elasticsearch. Tutaangalia ni mipangilio gani kwenye nodes inayohusika na usambazaji wa shards, jinsi ILM inavyofanya kazi na kufanya kazi.

Wale wanaofanya kazi na magogo, kwa njia moja au nyingine, wanakabiliwa na tatizo la kuhifadhi muda mrefu kwa uchambuzi wa baadaye. Katika Elasticsearch, hii ni kweli hasa, kwa sababu kila kitu kilikuwa na bahati mbaya na utendaji wa mtunza. Toleo la 6.6 lilianzisha utendakazi wa ILM. Inajumuisha awamu 4:

• Moto - Faharasa inasasishwa kikamilifu na kuulizwa.
• Joto - Faharasa haijasasishwa tena, lakini bado inaulizwa.
• Baridi - Faharasa haijasasishwa tena na mara chache hauulizwi. Taarifa lazima bado iweze kutafutwa, lakini hoja zinaweza kuwa polepole zaidi.
• Futa - Faharasa haihitajiki tena na inaweza kufutwa kwa usalama.

Imetolewa

• Elasticsearch Data Moto: vichakataji 24, kumbukumbu ya GB 128, 1,8 TB SSD RAID 10 (nodi 8).
• Elasticsearch Data Joto: vichakataji 24, kumbukumbu ya GB 64, Sera ya 8 ya TB NetApp SSD (nodi 4).
• Data ya Elasticsearch Baridi: Vichakataji 8, kumbukumbu ya GB 32, 128 TB HDD RAID 10 (nodi 4).

Lengo

Mipangilio hii ni ya mtu binafsi, yote inategemea mahali kwenye nodes, idadi ya indexes, magogo, nk. Tuna 2-3 TB ya data kwa siku.

• Siku 5 - Awamu ya moto (8 kuu / 1 replica).
• Siku 20 - awamu ya joto (shrink-index 4 kuu / 1 nakala).
• Siku 90 - awamu ya baridi (faharasa ya kufungia 4 kuu / 1 nakala).
• Siku 120 - Futa awamu.

Kuanzisha Elasticsearch

Ili kusambaza shards kwenye nodi, unahitaji parameta moja tu:

• Moto-nodi:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: hot

• Joto-nodi:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: warm

• Baridi-nodi:

  ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
  # Add custom attributes to the node:
  node.attr.box_type: cold

Kuanzisha Logstash

Yote hufanyaje kazi na jinsi gani tulitekeleza kipengele hiki? Wacha tuanze kwa kupata kumbukumbu kwenye Elasticsearch. Kuna njia mbili:

1. Logstash huchota kumbukumbu kutoka Kafka. Inaweza kuchukua safi au kubadilisha upande wako.
2. Kitu chenyewe kinaandika kwa Elasticsearch, kwa mfano, seva ya APM.

Fikiria mfano wa kudhibiti faharisi kupitia Logstash. Inaunda index na inatumika kwake muundo wa index na sambamba ILM.

k8s-ingress.conf

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # включаем управление шаблонами
        template_name => "k8s-ingress" # имя применяемого шаблона
        ilm_enabled => true # включаем управление ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи в индексы, должен быть уникальным
        ilm_pattern => "{now/d}-000001" # шаблон для создания индексов, может быть как "{now/d}-000001" так и "000001"
        ilm_policy => "k8s-ingress" # политика прикрепляемая к индексу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # название создаваемого индекса, может содержать %{+YYYY.MM.dd}, зависит от ilm_pattern
    }
}

Mpangilio wa Kibana

Kuna muundo msingi ambao unatumika kwa faharasa zote mpya. Inaweka usambazaji wa indexes za moto, idadi ya shards, replicas, nk. Uzito wa template imedhamiriwa na chaguo order. Violezo vilivyo na uzito wa juu hubatilisha vigezo vya violezo vilivyopo au kuongeza vipya.

PATA _kiolezo/chaguo-msingi

{
  "default" : {
    "order" : -1, # вес шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # применяем ко всем индексам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # уровень сжатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распределяем только по горячим нодам
            },
            "total_shards_per_node" : "8" # максимальное количество шардов на ноду от одного индекса
          }
        },
        "refresh_interval" : "5s", # интервал обновления индекса
        "number_of_shards" : "8", # количество шардов
        "auto_expand_replicas" : "0-1", # количество реплик на ноду от одного индекса
        "number_of_replicas" : "1" # количество реплик
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Kisha weka ramani kwenye faharasa k8s-ingress-* kwa kutumia template yenye uzito wa juu.

PATA _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Baada ya kutumia violezo vyote, tunatumia sera ya ILM na kuanza kufuatilia maisha ya faharasa.

PATA _ilm/policy/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # теплая фаза
          "min_age" : "5d", # срок жизни индекса после ротации до наступления теплой фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # куда перемещаем индекс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # обрезание индексов, т.к. у нас 4 ноды
            }
          }
        },
        "cold" : { # холодная фаза
          "min_age" : "25d", # срок жизни индекса после ротации до наступления холодной фазы
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # куда перемещаем индекс
              }
            },
            "freeze" : { } # замораживаем для оптимизации
          }
        },
        "hot" : { # горячая фаза
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # максимальный размер индекса до ротации (будет х2, т.к. есть 1 реплика)
              "max_age" : "1d" # максимальный срок жизни индекса до ротации
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # фаза удаления
          "min_age" : "120d", # максимальный срок жизни после ротации перед удалением
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Shida

Kulikuwa na matatizo katika hatua ya usanidi na utatuzi.

Awamu ya moto

Kwa mzunguko sahihi wa fahirisi, uwepo mwishoni ni muhimu index_name-date-000026 nambari za muundo 000001. Kuna mistari kwenye nambari inayoangalia faharisi kwa kutumia usemi wa kawaida wa uwepo wa nambari mwishoni. Vinginevyo, kutakuwa na hitilafu, hakuna sera zitatumika kwa index, na itakuwa daima katika awamu ya moto.

Awamu ya joto

Shrink (cutoff) - kupunguza idadi ya shards, kwa sababu tuna nodes 4 katika awamu ya joto na baridi. Nyaraka ina mistari ifuatayo:

• Faharasa lazima isomwe tu.
• Nakala ya kila shard kwenye faharasa lazima iwe kwenye nodi sawa.
• Hali ya afya ya nguzo lazima iwe ya kijani.

Ili kupogoa faharasa, Elasticsearch husogeza vijisehemu vyote vya msingi hadi kwenye nodi moja, na kurudia faharasa iliyopunguzwa na vigezo vinavyohitajika, na kisha kufuta ile ya zamani. Kigezo total_shards_per_node lazima iwe sawa na au kubwa zaidi ya idadi ya vipande vikuu vya kutoshea kwenye nodi moja. Vinginevyo, kutakuwa na arifa na shards haitahamia kwenye nodes sahihi.

PATA /shrink-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Awamu ya baridi

Kufungia (kufungia) - Tunafungia faharasa ili kuboresha hoja kwenye data ya kihistoria.

Utafutaji unaofanywa kwenye fahirisi zilizogandishwa hutumia uzi mdogo, uliojitolea, wa utafutaji ili kudhibiti idadi ya utafutaji unaofanana ambao hugusa vipande vilivyogandishwa kwenye kila nodi. Hii huweka mipaka ya kiasi cha kumbukumbu ya ziada inayohitajika kwa miundo ya data ya muda mfupi inayolingana na vipande vilivyogandishwa, ambayo kwa hivyo hulinda nodi dhidi ya utumizi mwingi wa kumbukumbu.
Fahirisi zilizogandishwa ni za kusoma pekee: huwezi kuziweka katika fahirisi.
Utafutaji kwenye fahirisi zilizogandishwa unatarajiwa kutekelezwa polepole. Fahirisi zilizogandishwa hazikusudiwa kupakia utafutaji wa juu. Inawezekana kwamba utafutaji wa faharasa uliogandishwa unaweza kuchukua sekunde au dakika kukamilika, hata kama utafutaji ule ule ulikamilika kwa milisekunde wakati fahirisi hazijagandishwa.

Matokeo ya

Tulijifunza jinsi ya kuandaa nodes za kufanya kazi na ILM, kuanzisha template ya kusambaza shards kati ya nodes za moto, na kuanzisha ILM kwa index na awamu zote za maisha.

Viungo muhimu

• https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-management-api.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/recovery-prioritization.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#indices-shrink-index
• https://www.elastic.co/guide/en/elasticsearch/reference/master/frozen-indices.html
• https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#shard-allocation-awareness

Chanzo: mapenzi.com