Voye Nginx json mòso bwa lè l sèvi avèk Vector nan Clickhouse ak Elasticsearch

Vektè, ki fèt pou kolekte, transfòme ak voye done boutèy demi lit, mezi ak evènman yo.

→ Github

Lè yo ekri nan lang Rust, li karakterize pa pèfòmans segondè ak konsomasyon RAM ki ba konpare ak analogue li yo. Anplis de sa, yo peye anpil atansyon sou fonksyon ki gen rapò ak kòrèkteman, an patikilye, kapasite nan sove evènman ki pa voye yo nan yon tanpon sou disk ak Thorne dosye.

Achitekti, Vector se yon routeur evènman ki resevwa mesaj ki soti nan youn oswa plis sous, opsyonèlman aplike sou mesaj sa yo transfòmasyon, epi voye yo bay youn oswa plis drenaj.

Vector se yon ranplasman pou filebeat ak logstash, li ka aji nan tou de wòl (resevwa epi voye mòso bwa), plis detay sou yo. Online.

Si nan Logstash chèn nan bati kòm opinyon → filtre → pwodiksyon Lè sa a, nan Vector li se sous → transfòme → lavabo

Egzanp yo ka jwenn nan dokiman an.

Enstriksyon sa a se yon enstriksyon revize soti nan Vyacheslav Rakhinsky. Enstriksyon orijinal yo gen pwosesis geoip. Lè tès geoip soti nan yon rezo entèn, vektè bay yon erè.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30

Si yon moun bezwen trete geoip, al gade nan enstriksyon orijinal yo soti nan Vyacheslav Rakhinsky.

Nou pral konfigirasyon konbinezon Nginx (Jounal Aksè) → Vektè (Kliyan | Filebeat) → Vektè (Sèvè | Logstash) → separeman nan Clickhouse ak separeman nan Elasticsearch. Nou pral enstale 4 serveurs. Malgre ke ou ka kontoune li ak 3 serveurs.

Konplo a se yon bagay tankou sa a.

Enfim Selinux sou tout serveurs ou yo

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

Nou enstale yon Emulation sèvè HTTP + sèvis piblik sou tout serveurs

Kòm yon Emulation sèvè HTTP nou pral itilize nodejs-souch-sèvè soti nan Maxim Ignatenko

Nodejs-stub-server pa gen yon rpm. Isit la kreye rpm pou li. rpm yo pral konpile lè l sèvi avèk Fedora Copr

Ajoute depo antonpatsev/nodejs-stub-server

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Enstale nodejs-stub-server, Apache benchmark ak ekran multiplexer tèminal sou tout serveurs

yum -y install stub_http_server screen mc httpd-tools screen

Mwen korije tan repons stub_http_server nan /var/lib/stub_http_server/stub_http_server.js fichye a pou te gen plis mòso bwa.

var max_sleep = 10;

Ann lanse stub_http_server.

systemctl start stub_http_server
systemctl enable stub_http_server

Enstalasyon Clickhouse sou sèvè 3

ClickHouse sèvi ak seri enstriksyon SSE 4.2 la, kidonk sof si yo espesifye otreman, sipò pou li nan processeur yo itilize a vin tounen yon kondisyon sistèm adisyonèl. Men lòd pou tcheke si processeur aktyèl la sipòte SSE 4.2:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Premye ou bezwen konekte depo ofisyèl la:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

Pou enstale pakè ou bezwen kouri kòmandman sa yo:

sudo yum install -y clickhouse-server clickhouse-client

Pèmèt clickhouse-server koute kat rezo a nan dosye /etc/clickhouse-server/config.xml

<listen_host>0.0.0.0</listen_host>

Chanje nivo anrejistreman an soti nan tras nan debogaj

dbug

Anviwònman konpresyon estanda:

min_compress_block_size  65536
max_compress_block_size  1048576

Pou aktive konpresyon Zstd, li te konseye pa manyen konfigirasyon an, men pito yo sèvi ak DDL.

Mwen pa t 'kapab jwenn ki jan yo sèvi ak konpresyon zstd atravè DDL nan Google. Se konsa, mwen kite li jan li ye.

Kòlèg ki itilize konpresyon zstd nan Clickhouse, tanpri pataje enstriksyon yo.

Pou kòmanse sèvè a kòm yon demon, kouri:

service clickhouse-server start

Koulye a, ann ale nan mete kanpe Clickhouse

Ale nan Clickhouse

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 — IP sèvè kote Clickhouse enstale.

Ann kreye yon baz done vektè

CREATE DATABASE vector;

Ann tcheke si baz done a egziste.

show databases;

Kreye yon tab vector.logs.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

Nou tcheke si tab yo te kreye. Ann lanse clickhouse-client epi fè yon demann.

Ann ale nan baz done vektè a.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Ann gade tab yo.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Enstale elastiksearch sou 4yèm sèvè a pou voye menm done yo bay Elasticsearch pou konparezon ak Clickhouse

Ajoute yon kle piblik rpm

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Ann kreye 2 repo:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Enstale elasticsearch ak kibana

yum install -y kibana elasticsearch

Piske li pral nan 1 kopi, ou bezwen ajoute sa ki annapre yo nan /etc/elasticsearch/elasticsearch.yml fichye a:

discovery.type: single-node

Se konsa, vektè sa a ka voye done nan elastiksearch soti nan yon lòt sèvè, ann chanje network.host.

network.host: 0.0.0.0

Pou konekte ak kibana, chanje paramèt server.host nan fichye /etc/kibana/kibana.yml.

server.host: "0.0.0.0"

Ansyen epi enkli elasticsearch nan autostart

systemctl enable elasticsearch
systemctl start elasticsearch

ak kibana

systemctl enable kibana
systemctl start kibana

Konfigirasyon Elasticsearch pou mòd yon sèl-nœud 1 shard, 0 kopi. Gen plis chans ou pral gen yon gwoup nan yon gwo kantite serveurs epi ou pa bezwen fè sa.

Pou pwochen endèks, mete ajou modèl default la:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' 

Enstalasyon Vektè kòm yon ranplasman pou Logstash sou sèvè 2

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Ann mete Vector kòm yon ranplasman pou Logstash. Modification fichye /etc/vector/vector.toml

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

Ou ka ajiste seksyon transforms.nginx_parse_add_defaults.

Kòm Vyacheslav Rakhinsky sèvi ak konfigirasyon sa yo pou yon ti CDN epi ka gen plizyè valè nan upstream_*

Pou egzanp:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Si sa a se pa sitiyasyon ou, Lè sa a, seksyon sa a ka senplifye

Ann kreye anviwònman sèvis pou systemd /etc/systemd/system/vector.service

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Apre ou fin kreye tab yo, ou ka kouri Vector

systemctl enable vector
systemctl start vector

Jounal vektè yo ka wè jan sa a:

journalctl -f -u vector

Ta dwe gen antre tankou sa a nan mòso bwa yo

INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.

Sou kliyan an (sèvè Web) - 1ye sèvè

Sou sèvè a ak nginx, ou bezwen enfim ipv6, depi tab mòso bwa a nan clickhouse sèvi ak jaden an. upstream_addr IPv4, depi mwen pa sèvi ak ipv6 andedan rezo a. Si ipv6 pa etenn, pral gen erè:

DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)

Petèt lektè yo, ajoute sipò ipv6.

Kreye fichye /etc/sysctl.d/98-disable-ipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Aplike paramèt yo

sysctl --system

Ann enstale nginx.

Te ajoute dosye depo nginx /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Enstale pake nginx la

yum install -y nginx

Premyèman, nou bezwen konfigirasyon fòma boutèy la nan Nginx nan dosye /etc/nginx/nginx.conf.

user  nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Nan lòd pa kraze konfigirasyon ou ye kounye a, Nginx pèmèt ou gen plizyè direktiv access_log

access_log  /var/log/nginx/access.log  main;            # Стандартный лог
access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

Pa bliye ajoute yon règ nan logrotate pou nouvo mòso bwa (si dosye a pa fini ak .log)

Retire default.conf nan /etc/nginx/conf.d/

rm -f /etc/nginx/conf.d/default.conf

Ajoute lame vityèl /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Ajoute lame vityèl /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Ajoute lame vityèl /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Ajoute lame vityèl /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Ajoute lame vityèl (172.26.10.106 ip nan sèvè a kote nginx enstale) nan tout serveurs nan /etc/hosts fichye a:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Men, si tout bagay pare lè sa a

nginx -t 
systemctl restart nginx

Koulye a, ann enstale li tèt nou Vektè

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Ann kreye yon dosye paramèt pou systemd /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Epi konfigirasyon ranplasman Filebeat nan /etc/vector/vector.toml konfigirasyon an. Adrès IP 172.26.10.108 se adrès IP sèvè boutèy la (Vektè-Sèvè)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Pa bliye ajoute itilizatè vektè a nan gwoup obligatwa a pou li ka li dosye log. Pou egzanp, nginx nan centos kreye mòso bwa ak dwa gwoup adm.

usermod -a -G adm vector

Ann kòmanse sèvis vektè a

systemctl enable vector
systemctl start vector

Jounal vektè yo ka wè jan sa a:

journalctl -f -u vector

Ta dwe gen yon antre tankou sa a nan mòso bwa yo

INFO vector::topology::builder: Healthcheck: Passed.

Tès estrès

Nou fè tès lè l sèvi avèk Apache referans.

Pake httpd-tools la te enstale sou tout serveurs

Nou kòmanse fè tès lè l sèvi avèk Apache referans soti nan 4 serveurs diferan nan ekran. Premyèman, nou lanse multiplexeur tèminal ekran an, ak Lè sa a, nou kòmanse tès lè l sèvi avèk referans Apache la. Ki jan yo travay ak ekran ou ka jwenn nan Atik.

Soti nan 1ye sèvè

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Soti nan 2ye sèvè

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Soti nan 3ye sèvè

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Soti nan 4ye sèvè

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Ann tcheke done yo nan Clickhouse

Ale nan Clickhouse

clickhouse-client -h 172.26.10.109 -m

Fè yon rechèch SQL

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴──────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Chèche konnen gwosè tab yo nan Clickhouse

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Ann chèche konnen ki kantite mòso bwa te pran nan Clickhouse.

Gwosè tab mòso bwa a se 857.19 MB.

Gwosè a nan menm done yo nan endèks la nan Elasticsearch se 4,5GB.

Si ou pa presize done nan vektè a nan paramèt yo, Clickhouse pran 4500/857.19 = 5.24 fwa mwens pase nan Elasticsearch.

Nan vektè, se jaden an konpresyon yo itilize pa default.

Telegram chat pa Clickhouse
Telegram chat pa Elasticsearch
Telegram chat pa "Koleksyon ak analiz de sistèm mesaj"

Sous: www.habr.com