Kutumira Nginx json matanda uchishandisa Vector kuClickhouse uye Elasticsearch

Vector, yakagadzirirwa kuunganidza, kushandura uye kutumira log data, metrics uye zviitiko.

→ Github

Kunyorwa mumutauro weRust, inoratidzirwa nekuita kwepamusoro uye kuderera kwe RAM kushandiswa kana ichienzaniswa nemaanalogues ayo. Uye zvakare, kutarisisa kwakanyanya kunobhadharwa kumabasa ane chekuita nekurongeka, kunyanya, kugona kuchengetedza zviitiko zvisina kutumirwa kune buffer pa diski uye kutenderera mafaera.

Architecturally, Vector ndeye chiitiko router inogamuchira mameseji kubva kune imwe kana kupfuura zvinyorwa, nekusarudza kushandisa pamusoro pemameseji aya shanduko, uye kuvatumira kune mumwe kana kupfuura drains.

Vector inotsiva filebeat uye logstash, inogona kuita muzvikamu zviviri (kugamuchira uye kutumira matanda), zvimwe zvakawanda pazviri. site.

Kana muLogstash ketani inovakwa seyekupinza → sefa → kubuda ipapo muVector ndizvo zvitubu → shanduko → anonyura

Mienzaniso inogona kuwanikwa mune zvinyorwa.

Murairo uyu murairo wakadzokororwa kubva Vyacheslav Rakhinsky. Mirayiridzo yepakutanga ine geoip processing. Paunenge uchiyedza geoip kubva kunetiweki yemukati, vector yakapa kukanganisa.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30

Kana paine anoda kugadzirisa geoip, wobva watarisa kune yekutanga mirairo kubva Vyacheslav Rakhinsky.

Isu tichagadzirisa musanganiswa weNginx (Kupinda matanda) → Vector (Mutengi | Filebeat) → Vector (Server | Logstash) → zvakasiyana muClickhouse uye zvakasiyana muElasticsearch. Tichaisa maseva mana. Kunyangwe iwe uchigona kuipfuura ne4 maseva.

The scheme chinhu chakadai.

Dzima Selinux pamaseva ako ese

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

Isu tinoisa HTTP sevha emulator + zvishandiso pane ese maseva

Seye HTTP sevha emulator isu tichashandisa nodejs-stub-server от Maxim Ignatenko

Nodejs-stub-server haina rpm. zviri gadzira rpm kwayo. rpm ichaunganidzwa uchishandisa Fedora Copr

Wedzera iyo antonpatsev/nodejs-stub-server repository

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Isa nodejs-stub-server, Apache bhenji uye screen terminal multiplexer pane ese maseva

yum -y install stub_http_server screen mc httpd-tools screen

Ndakagadzirisa stub_http_server nguva yekupindura mu /var/lib/stub_http_server/stub_http_server.js faira kuitira kuti pave nemamwe matanda.

var max_sleep = 10;

Ngatitangei stub_http_server.

systemctl start stub_http_server
systemctl enable stub_http_server

Clickhouse installation pa server 3

ClickHouse inoshandisa iyo SSE 4.2 yekuraira seti, saka kunze kwekunge yatsanangurwa neimwe nzira, tsigiro yayo mu processor inoshandiswa inova imwe yekuwedzera system inodiwa. Heino murairo wekutarisa kana processor iripo inotsigira SSE 4.2:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Kutanga iwe unofanirwa kubatanidza iyo official repository:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

Kuisa mapakeji iwe unofanirwa kumhanyisa inotevera mirairo:

sudo yum install -y clickhouse-server clickhouse-client

Bvumira clickhouse-server kuti iteerere kunetiweki kadhi mufaira /etc/clickhouse-server/config.xml

<listen_host>0.0.0.0</listen_host>

Kusandura nhanho yekutema kubva pakuteedzera kuenda kune debug

debug

Standard compression settings:

min_compress_block_size  65536
max_compress_block_size  1048576

Kuti uvhure Zstd compression, yakarairwa kuti isabate config, asi kushandisa DDL.

Handina kuwana nzira yekushandisa zstd compression kuburikidza neDDL muGoogle. Saka ndakasiya zvakadaro.

Shamwari dzinoshandisa zstd compression muClickhouse, ndapota govera mirairo.

Kutanga sevha se daemon, mhanya:

service clickhouse-server start

Zvino ngatienderere mberi kumisikidza Clickhouse

Enda kuClickhouse

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 - IP yevhavha iyo Clickhouse yakaiswa.

Ngatigadzirei vector database

CREATE DATABASE vector;

Ngatitarisei kuti database iripo.

show databases;

Gadzira tafura yevector.logs.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

Tinotarisa kuti matafura akagadzirwa. Ngatitangei clickhouse-client uye ita chikumbiro.

Ngatiende kune vector database.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Ngatitarisei pamatafura.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Kuisa elasticsearch pane 4th server kutumira iyo yakafanana data kuElasticsearch yekuenzanisa neClickhouse.

Wedzera kiyi yeruzhinji rpm

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Ngatigadzirei 2 repo:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Isa elasticsearch uye kibana

yum install -y kibana elasticsearch

Sezvo ichange iri mukopi imwe, unofanirwa kuwedzera zvinotevera kune /etc/elasticsearch/elasticsearch.yml faira:

discovery.type: single-node

Saka kuti vector inogona kutumira data kune elasticsearch kubva kune imwe sevha, ngatichinje network.host.

network.host: 0.0.0.0

Kuti ubatanidze kukibana, shandura iyo server.host parameter mufaira /etc/kibana/kibana.yml

server.host: "0.0.0.0"

Yekare uye inosanganisira elasticsearch mu autostart

systemctl enable elasticsearch
systemctl start elasticsearch

uye kibana

systemctl enable kibana
systemctl start kibana

Kugadzirisa Elasticsearch yeimwe-node modhi 1 shard, 0 replica. Zvingangodaro iwe uchave nesumbu renhamba huru yemaseva uye haufanirwe kuita izvi.

Kuti uwane ma indexes emangwana, gadziridza iyo default template:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' 

Kuiswa Vector sekutsiva kweLogstash pane server 2

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Ngatimisei Vector sechinotsiva Logstash. Kugadzirisa faira /etc/vector/vector.toml

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

Unogona kugadzirisa transforms.nginx_parse_add_defaults chikamu.

kubva Vyacheslav Rakhinsky inoshandisa izvi zvigadziriso zveCDN diki uye panogona kuve nemaitiro akati wandei kumusoro kumusoro_*

Somuenzaniso:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Kana aya asiri mamiriro ako, saka chikamu ichi chinogona kurerutswa

Ngatigadzirei kuseta masevhisi e systemd /etc/systemd/system/vector.service

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Mushure mekugadzira matafura, unogona kumhanya Vector

systemctl enable vector
systemctl start vector

Vector logs inogona kutariswa seizvi:

journalctl -f -u vector

Panofanira kunge paine zvinyorwa zvakaita seizvi mumatanda

INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.

Pamutengi (Web server) - 1st server

Pane sevha ine nginx, unofanirwa kudzima ipv6, sezvo tafura yematanda mu clickhouse inoshandisa munda. upstream_addr IPv4, sezvo ini ndisingashandisi ipv6 mukati metiweki. Kana ipv6 isina kudzimwa, pachave nezvikanganiso:

DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)

Zvichida vaverengi, wedzera ipv6 rutsigiro.

Gadzira faira /etc/sysctl.d/98-disable-ipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Kushandisa marongero

sysctl --system

Ngatiise nginx.

Yakawedzera nginx repository faira /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Isa iyo nginx package

yum install -y nginx

Kutanga, isu tinofanirwa kugadzirisa iyo log format muNginx mufaira /etc/nginx/nginx.conf

user  nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Kuti usatyore gadziriso yako yazvino, Nginx inobvumidza iwe kuve neakati wandei access_log dhairekitori

access_log  /var/log/nginx/access.log  main;            # Стандартный лог
access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

Usakanganwa kuwedzera mutemo wekutora matanda matsva (kana irogi faira risingapere ne.log)

Bvisa default.conf kubva /etc/nginx/conf.d/

rm -f /etc/nginx/conf.d/default.conf

Wedzera virtual host /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Wedzera virtual host /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Wedzera virtual host /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Wedzera virtual host /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Wedzera mauto chaiwo (172.26.10.106 ip yevhavha iyo nginx yakaiswa) kune ese maseva kune /etc/hosts faira:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Uye kana zvose zvagadzirira ipapo

nginx -t 
systemctl restart nginx

Zvino ngatiiise isu pachedu Vector

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Ngatigadzire faira rekuisa systemd /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Uye gadzirisa iyo Filebeat inotsiva mu /etc/vector/vector.toml config. IP kero 172.26.10.108 ndiyo IP kero yelog server (Vector-Server)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Usakanganwa kuwedzera vector mushandisi kuboka rinodiwa kuti averenge mafaira egi. Semuenzaniso, nginx mu centos inogadzira matanda ane adm boka kodzero.

usermod -a -G adm vector

Ngatitange basa revector

systemctl enable vector
systemctl start vector

Vector logs inogona kutariswa seizvi:

journalctl -f -u vector

Panofanira kuva nekupinda seizvi mumatanda

INFO vector::topology::builder: Healthcheck: Passed.

Stress Testing

Isu tinoita bvunzo tichishandisa Apache bhenji.

Iyo httpd-zvishandiso package yakaiswa pane ese maseva

Isu tinotanga kuyedza tichishandisa Apache bhenji kubva kune mana akasiyana maseva muchiratidziro. Kutanga, isu tinotangisa iyo screen terminal multiplexer, uye tobva tatanga kuyedza tichishandisa iyo Apache bhenji. Maitiro ekushanda nescreen iwe unogona kuwana mukati chinyorwa.

Kubva 1st server

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Kubva 2st server

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Kubva 3st server

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Kubva 4st server

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Ngatitarisei iyo data muClickhouse

Enda kuClickhouse

clickhouse-client -h 172.26.10.109 -m

Kuita mubvunzo weSQL

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴──────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Tsvaga saizi yematafura muClickhouse

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Ngationei kuti ingani matanda akatora muClickhouse.

Iyo matanda tafura saizi ndeye 857.19 MB.

Saizi ye data yakafanana mune index muElasticsearch ndeye 4,5GB.

Kana iwe ukasatsanangura data muVector mune ma paramita, Clickhouse inotora 4500/857.19 = 5.24 nguva shoma pane muElasticsearch.

Mune vector, iyo compression munda inoshandiswa nekukasira.

Teregiramu chat by Clickhouse
Teregiramu chat by Elasticsearch
Telegraph chat by "Kuunganidza uye kuongororwa kwehurongwa mameseji"

Source: www.habr.com