ProHoster > Π‘Π»ΠΎΠ³ > administratio > Diu terminus notitia repono in Elasticsearch

Diu terminus notitia repono in Elasticsearch

Diu terminus notitia repono in Elasticsearch

Meum nomen est Igor Sidorenko, technicus dux sum in turma adminorum qui totam infrastructuram Domclick conservant.

Experientiam meam communicare cupimus in repositione distributa data in elastica investigatione constituenda. Videbimus quid occasus in nodis respondeant distributioni shards, quomodo ILM opera et opera.

Illi qui lignis operantur, modo vel alio, quaestionem longi temporis repositionis in posterioribus analysi laborant. In Elastica investigatione, hoc maxime verum est, quia omnia ad munus curatoris infortunati sunt. Versio 6.6 induxit ILM functionality. Constat ex 4 gradibus;

  • Hot - Index active renovatur et queritur.
  • Calidum - Index non iam renovatur, sed adhuc queritur.
  • Frigus - index non amplius renovatur ac raro queritur. Indicium adhuc perscrutabile est, sed quaestionibus tardius fieri potest.
  • Delere - Index iam non indiget et tuto deletum est.

Datum

  • Elastica inquisitionis Data Hot: 24 processors, 128 GB memoria, 1,8 TB SSD RAID 10 (8 nodi).
  • Elastica inquisitionis Data Warm: 24 processores, 64 GB memoria, 8 TB NetApp SSD Policy (4 nodi).
  • Elastica inquisitionis Data Frigus: 8 processores, 32 GB memoria, 128 TB HDD RAID 10 (4 nodi).

propositum

Hae uncinis singulae sunt, totum pendet a loco in nodis, indicibus, lignis, etc. Habemus 2-3 TB notitiarum per diem.

  • 5 dies - Hot phase (8 main / 1 replica).
  • XX dies - calidum tempus (reformidant-index 4 principalis / 1 replica).
  • XC dies - Tempus Frigidum (Frigidus-index 4 principalis / 1 replica).
  • CXX dies - Delere pascha.

Elasticsearch profecta sunt

Ligulas trans nodi distribuas, modulo uno tantum indiges;

  • Hot-nodes: 
    ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: hot
  • Calida-nodes: 
    ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: warm
  • frigus-nodes: 
    ~]# cat /etc/elasticsearch/elasticsearch.yml | grep attr
# Add custom attributes to the node:
node.attr.box_type: cold

Occasus sursum Logstash

Quomodo totum opus facit et quomodo hanc plumam efficiendum fecimus? Incipiamus per questus omnia in Elasticsearch. Dupliciter;

  1. Logstash ligna ex Kafka accersit. Potest colligere munda vel in latus tuum convertere.
  2. Aliquid ipse scribit ad Elastica inquisitionem, exempli gratia, servo APM.

Exemplum habe index administrandi per Logstash. Is index gignit et applicat index exemplaris et correspondentes ILM.

k8s-ingress.conf

input {
    kafka {
        bootstrap_servers => "node01, node02, node03"
        topics => ["ingress-k8s"]
        decorate_events => false
        codec => "json"
    }
}

filter {
    ruby {
        path => "/etc/logstash/conf.d/k8s-normalize.rb"
    }
    if [log] =~ "[warn]" or [log] =~ "[error]" or [log] =~ "[notice]" or [log] =~ "[alert]" {
        grok {
            match => { "log" => "%{DATA:[nginx][error][time]} [%{DATA:[nginx][error][level]}] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: *%{NUMBER:[nginx][error][connection_id]} %{DATA:[nginx][error][message]}, client: %{IPORHOST:[nginx][error][remote_ip]}, server: %{DATA:[nginx][error][server]}, request: "%{WORD:[nginx][error][method]} %{DATA:[nginx][error][url]} HTTP/%{NUMBER:[nginx][error][http_version]}", (?:upstream: "%{DATA:[nginx][error][upstream][proto]}://%{DATA:[nginx][error][upstream][host]}:%{DATA:[nginx][error][upstream][port]}/%{DATA:[nginx][error][upstream][url]}", )?host: "%{DATA:[nginx][error][host]}"(?:, referrer: "%{DATA:[nginx][error][referrer]}")?" }
            remove_field => "log"
        }
    }
    else {
        grok {
            match => { "log" => "%{IPORHOST:[nginx][access][host]} - [%{IPORHOST:[nginx][access][remote_ip]}] - %{DATA:[nginx][access][remote_user]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][bytes_sent]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" %{NUMBER:[nginx][access][request_lenght]} %{NUMBER:[nginx][access][request_time]} [%{DATA:[nginx][access][upstream][name]}] (?:-|%{IPORHOST:[nginx][access][upstream][addr]}:%{NUMBER:[nginx][access][upstream][port]}) (?:-|%{NUMBER:[nginx][access][upstream][response_lenght]}) %{DATA:[nginx][access][upstream][response_time]} %{DATA:[nginx][access][upstream][status]} %{DATA:[nginx][access][request_id]}" }
            remove_field => "log"
        }
    }
}
output {
    elasticsearch {
        id => "k8s-ingress"
        hosts => ["node01", "node02", "node03", "node04", "node05", "node06", "node07", "node08"]
        manage_template => true # Π²ΠΊΠ»ΡŽΡ‡Π°Π΅ΠΌ ΡƒΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ шаблонами
        template_name => "k8s-ingress" # имя примСняСмого шаблона
        ilm_enabled => true # Π²ΠΊΠ»ΡŽΡ‡Π°Π΅ΠΌ ΡƒΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ILM
        ilm_rollover_alias => "k8s-ingress" # alias для записи Π² индСксы, Π΄ΠΎΠ»ΠΆΠ΅Π½ Π±Ρ‹Ρ‚ΡŒ ΡƒΠ½ΠΈΠΊΠ°Π»ΡŒΠ½Ρ‹ΠΌ
        ilm_pattern => "{now/d}-000001" # шаблон для создания индСксов, ΠΌΠΎΠΆΠ΅Ρ‚ Π±Ρ‹Ρ‚ΡŒ ΠΊΠ°ΠΊ "{now/d}-000001" Ρ‚Π°ΠΊ ΠΈ "000001"
        ilm_policy => "k8s-ingress" # ΠΏΠΎΠ»ΠΈΡ‚ΠΈΠΊΠ° прикрСпляСмая ΠΊ индСксу
        index => "k8s-ingress-%{+YYYY.MM.dd}" # Π½Π°Π·Π²Π°Π½ΠΈΠ΅ создаваСмого индСкса, ΠΌΠΎΠΆΠ΅Ρ‚ ΡΠΎΠ΄Π΅Ρ€ΠΆΠ°Ρ‚ΡŒ %{+YYYY.MM.dd}, зависит ΠΎΡ‚ ilm_pattern
    }
}

Kibana setup

Basis est forma quae ad omnes indices novos applicatur. Distributio indices calidas, numerum testae, replicas, etc. Formulae pondus optionis determinatur order. Templates cum altiore pondere dominari parametros templates existentium vel novas addere.

Diu terminus notitia repono in Elasticsearch
Diu terminus notitia repono in Elasticsearch

GET_template/default

{
  "default" : {
    "order" : -1, # вСс шаблона
    "version" : 1,
    "index_patterns" : [
      "*" # примСняСм ΠΊΠΎ всСм индСксам
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression", # ΡƒΡ€ΠΎΠ²Π΅Π½ΡŒ сТатия
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot" # распрСдСляСм Ρ‚ΠΎΠ»ΡŒΠΊΠΎ ΠΏΠΎ горячим Π½ΠΎΠ΄Π°ΠΌ
            },
            "total_shards_per_node" : "8" # максимальноС количСство ΡˆΠ°Ρ€Π΄ΠΎΠ² Π½Π° Π½ΠΎΠ΄Ρƒ ΠΎΡ‚ ΠΎΠ΄Π½ΠΎΠ³ΠΎ индСкса
          }
        },
        "refresh_interval" : "5s", # ΠΈΠ½Ρ‚Π΅Ρ€Π²Π°Π» обновлСния индСкса
        "number_of_shards" : "8", # количСство ΡˆΠ°Ρ€Π΄ΠΎΠ²
        "auto_expand_replicas" : "0-1", # количСство Ρ€Π΅ΠΏΠ»ΠΈΠΊ Π½Π° Π½ΠΎΠ΄Ρƒ ΠΎΡ‚ ΠΎΠ΄Π½ΠΎΠ³ΠΎ индСкса
        "number_of_replicas" : "1" # количСство Ρ€Π΅ΠΏΠ»ΠΈΠΊ
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { }
    },
    "aliases" : { }
  }
}

Applicare deinde tabularum faciendarum ad indices k8s-ingress-* salvis utens cum superiore pondere.

Diu terminus notitia repono in Elasticsearch
Diu terminus notitia repono in Elasticsearch

GET _template/k8s-ingress

{
  "k8s-ingress" : {
    "order" : 100,
    "index_patterns" : [
      "k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "hot"
            }
          }
        },
        "number_of_shards" : "8",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "numeric_detection" : false,
      "_meta" : { },
      "_source" : { },
      "dynamic_templates" : [
        {
          "all_fields" : {
            "mapping" : {
              "index" : false,
              "type" : "text"
            },
            "match" : "*"
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "kubernetes" : {
          "type" : "object",
          "properties" : {
            "container_name" : {
              "type" : "keyword"
            },
            "container_hash" : {
              "index" : false,
              "type" : "keyword"
            },
            "host" : {
              "type" : "keyword"
            },
            "annotations" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "index" : false,
                  "type" : "text"
                },
                "key" : {
                  "index" : false,
                  "type" : "keyword"
                }
              }
            },
            "docker_id" : {
              "index" : false,
              "type" : "keyword"
            },
            "pod_id" : {
              "type" : "keyword"
            },
            "labels" : {
              "type" : "object",
              "properties" : {
                "value" : {
                  "type" : "keyword"
                },
                "key" : {
                  "type" : "keyword"
                }
              }
            },
            "namespace_name" : {
              "type" : "keyword"
            },
            "pod_name" : {
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "nginx" : {
          "type" : "object",
          "properties" : {
            "access" : {
              "type" : "object",
              "properties" : {
                "agent" : {
                  "type" : "text"
                },
                "response_code" : {
                  "type" : "integer"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    },
                    "response_lenght" : {
                      "type" : "integer"
                    },
                    "response_time" : {
                      "index" : false,
                      "type" : "text"
                    },
                    "addr" : {
                      "type" : "keyword"
                    },
                    "status" : {
                      "index" : false,
                      "type" : "text"
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "bytes_sent" : {
                  "type" : "integer"
                },
                "request_lenght" : {
                  "type" : "integer"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "remote_user" : {
                  "type" : "text"
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "request_time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            },
            "error" : {
              "type" : "object",
              "properties" : {
                "server" : {
                  "type" : "keyword"
                },
                "upstream" : {
                  "type" : "object",
                  "properties" : {
                    "port" : {
                      "type" : "keyword"
                    },
                    "proto" : {
                      "type" : "keyword"
                    },
                    "host" : {
                      "type" : "keyword"
                    },
                    "url" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword"
                        }
                      }
                    }
                  }
                },
                "method" : {
                  "type" : "keyword"
                },
                "level" : {
                  "type" : "keyword"
                },
                "http_version" : {
                  "type" : "keyword"
                },
                "pid" : {
                  "index" : false,
                  "type" : "integer"
                },
                "message" : {
                  "type" : "text"
                },
                "tid" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword"
                    }
                  }
                },
                "referrer" : {
                  "type" : "text"
                },
                "remote_ip" : {
                  "type" : "ip"
                },
                "connection_id" : {
                  "index" : false,
                  "type" : "keyword"
                },
                "host" : {
                  "type" : "keyword"
                },
                "time" : {
                  "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis||dd/MMM/YYYY:H:m:s Z",
                  "type" : "date"
                }
              }
            }
          }
        },
        "log" : {
          "type" : "text"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "eventtime" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
  }
}

Post omnia exempla applicanda, consilium ILM applicamus et vita indicium vigilantia incipimus.

Diu terminus notitia repono in Elasticsearch

Diu terminus notitia repono in Elasticsearch

Diu terminus notitia repono in Elasticsearch

ADEPTO _ilm/consilium/k8s-ingress

{
  "k8s-ingress" : {
    "version" : 14,
    "modified_date" : "2020-06-11T10:27:01.448Z",
    "policy" : {
      "phases" : {
        "warm" : { # тСплая Ρ„Π°Π·Π°
          "min_age" : "5d", # срок ΠΆΠΈΠ·Π½ΠΈ индСкса послС Ρ€ΠΎΡ‚Π°Ρ†ΠΈΠΈ Π΄ΠΎ наступлСния Ρ‚Π΅ΠΏΠ»ΠΎΠΉ Ρ„Π°Π·Ρ‹
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm" # ΠΊΡƒΠ΄Π° ΠΏΠ΅Ρ€Π΅ΠΌΠ΅Ρ‰Π°Π΅ΠΌ индСкс
              }
            },
            "shrink" : {
              "number_of_shards" : 4 # ΠΎΠ±Ρ€Π΅Π·Π°Π½ΠΈΠ΅ индСксов, Ρ‚.ΠΊ. Ρƒ нас 4 Π½ΠΎΠ΄Ρ‹
            }
          }
        },
        "cold" : { # холодная Ρ„Π°Π·Π°
          "min_age" : "25d", # срок ΠΆΠΈΠ·Π½ΠΈ индСкса послС Ρ€ΠΎΡ‚Π°Ρ†ΠΈΠΈ Π΄ΠΎ наступлСния Ρ…ΠΎΠ»ΠΎΠ΄Π½ΠΎΠΉ Ρ„Π°Π·Ρ‹
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold" # ΠΊΡƒΠ΄Π° ΠΏΠ΅Ρ€Π΅ΠΌΠ΅Ρ‰Π°Π΅ΠΌ индСкс
              }
            },
            "freeze" : { } # Π·Π°ΠΌΠΎΡ€Π°ΠΆΠΈΠ²Π°Π΅ΠΌ для ΠΎΠΏΡ‚ΠΈΠΌΠΈΠ·Π°Ρ†ΠΈΠΈ
          }
        },
        "hot" : { # горячая Ρ„Π°Π·Π°
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb", # ΠΌΠ°ΠΊΡΠΈΠΌΠ°Π»ΡŒΠ½Ρ‹ΠΉ Ρ€Π°Π·ΠΌΠ΅Ρ€ индСкса Π΄ΠΎ Ρ€ΠΎΡ‚Π°Ρ†ΠΈΠΈ (Π±ΡƒΠ΄Π΅Ρ‚ Ρ…2, Ρ‚.ΠΊ. Π΅ΡΡ‚ΡŒ 1 Ρ€Π΅ΠΏΠ»ΠΈΠΊΠ°)
              "max_age" : "1d" # ΠΌΠ°ΠΊΡΠΈΠΌΠ°Π»ΡŒΠ½Ρ‹ΠΉ срок ΠΆΠΈΠ·Π½ΠΈ индСкса Π΄ΠΎ Ρ€ΠΎΡ‚Π°Ρ†ΠΈΠΈ
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : { # Ρ„Π°Π·Π° удалСния
          "min_age" : "120d", # ΠΌΠ°ΠΊΡΠΈΠΌΠ°Π»ΡŒΠ½Ρ‹ΠΉ срок ΠΆΠΈΠ·Π½ΠΈ послС Ρ€ΠΎΡ‚Π°Ρ†ΠΈΠΈ ΠΏΠ΅Ρ€Π΅Π΄ ΡƒΠ΄Π°Π»Π΅Π½ΠΈΠ΅ΠΌ
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

problems

Problemata erant in spectaculo et scaena debugging.

Calidum tempus

Recta enim indices gyratio, praesentia in fine critica est index_name-date-000026 forma numeri 000001. Sunt lineae in codice qui indices reprimunt adhibitis expressionibus regularibus ad praesentiam numerorum in fine. Alioquin error erit, nullae rationes ad indicem adhibebuntur, et semper in tempore calido erit.

Calidum tempus

shrink - reducendo numerum shardorum, quia habemus 4 nodos in augmentis calidis et frigidis. Documenta sequentes lineas continet:

  • Index legi debet-tantum.
  • Exemplar cuiuslibet testae in indice in eodem nodo residere debet.
  • Botrus sanitatis viridis esse debet.

Ad indicem putandum, elasticae inquisitionis primas omnes ad unum nodi shards movet, indicem truncatum cum parametris necessariis duplicat, ac deinde vetus delet. Parameter total_shards_per_node aequale vel maius esse debet quam numerus principalis shards in unum nodi aptare. Alioquin notificationes et shards ad nodos rectos non movebuntur.

Diu terminus notitia repono in Elasticsearch
Diu terminus notitia repono in Elasticsearch

GET/horreat-k8s-ingress-2020.06.06-000025/_settings

{
  "shrink-k8s-ingress-2020.06.06-000025" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "shrink-k8s-ingress-2020.06.06-000025",
        "creation_date" : "1592225525569",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "psF4MiFGQRmi8EstYUQS4w",
        "version" : {
          "created" : "7060299",
          "upgraded" : "7060299"
        },
        "lifecycle" : {
          "name" : "k8s-ingress",
          "rollover_alias" : "k8s-ingress",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "initial_recovery" : {
              "_id" : "_Le0Ww96RZ-o76bEPAWWag"
            },
            "require" : {
              "_id" : null,
              "box_type" : "cold"
            },
            "total_shards_per_node" : "8"
          }
        },
        "number_of_shards" : "4",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : "k8s-ingress-2020.06.06-000025",
            "uuid" : "gNhYixO6Skqi54lBjg5bpQ"
          }
        }
      }
    }
  }
}

Frigus tempus

Spermatophyta (Frigidus) - Frigidus indicem ad quaestiones optimize in notitia historica.

Investigationes factae in indicibus congelatis utantur parvis, dedicatis, search__fixis stamina ad refrenandum numerum inquisitionum concurrentium quae in singulis nodi shards congelatas feriunt. Hoc limites moles extra memoriam requisita est pro structuris transeuntibus shards congelatis respondens, quae proinde nodos contra nimiam memoriam consummationis tutatur.
Indices congelati legunt solum: in eos index non potes.
Investigationes in indicibus congelatis exspectantur ut tardius exequantur. Indices congelati non sunt destinati ad onus altum inquisitionis. Fieri potest ut perquisitio indicis glacialis secunda vel minuta ad perficiendum capere possit, etiamsi eaedem inquisitiones in milliseconds cum indicibus non congelatis compleantur.

results

Nodos parandi ad operandum cum ILM discimus, exemplum schedulae distribuendae inter lymphas nodos posui, et ILM ad indicem cum omnibus vitae gradibus constituimus.

Nexus utilis

Source: www.habr.com