Têketinên Nginx json bi karanîna Vector ji Clickhouse û Elasticsearch re dişînin

Vektor, ji bo berhevkirin, veguherandin û şandina daneyên têketinê, metrîk û bûyeran hatî çêkirin.

→ Github

Ji ber ku bi zimanê Rust hatî nivîsandin, li gorî analogên xwe bi performansa bilind û xerckirina kêm RAM-ê tê destnîşan kirin. Digel vê yekê, pir bal tê kişandin ser fonksiyonên ku bi rastbûnê ve girêdayî ne, nemaze, şiyana hilanîna bûyerên neşandiye li tamponek li ser dîskê û zivirîna pelan.

Ji hêla mîmarî ve, Vector routerek bûyerê ye ku ji yek an jî bêtir peyaman distîne çavkaniyên, vebijarkî li ser van peyaman sepandin veguherînan, û şandina wan ji yek an jî zêdetir dirijîne.

Vektor ji bo pelêbeat û logstash-ê veguhezek e, ew dikare di her du rolan de tevbigere (qeydên wergirtin û şandin), bêtir hûrguliyên li ser wan malperê.

Ger di Logstash de zincîre wekî têketin → Parzûn → derketin çêbibe wê hingê ew di Vector de ye çavkaniyên → veguherîne → diherikî

Nimûneyên di belgeyê de têne dîtin.

Ev talîmat talîmatek revîzekirî ye ji Vyacheslav Rakhinsky. Rêbernameyên orîjînal pêvajoyek geoip vedihewîne. Dema ceribandina geoip ji torgilokek navxweyî, vektor xeletiyek da.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30

Ger kesek hewce bike ku geoip pêvajoyê bike, wê hingê serî li rêwerzên orîjînal ên ji Vyacheslav Rakhinsky.

Em ê lihevhatina Nginx (qeydên gihîştinê) → Vektor (Xerîdar | Pelêbeat) → Vektor (Server | Logstash) → veqetandî di Clickhouse de û ji hev veqetandî di Elasticsearch de mîheng bikin. Em ê 4 pêşkêşkeran saz bikin. Her çend hûn dikarin wê bi 3 serveran re derbas bikin.

Plan tiştekî bi vî rengî ye.

Selinux li ser hemî serverên xwe neçalak bikin

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

Em li ser hemî serveran emulatorek servera HTTP + karûbar saz dikin

Em ê wekî emûlatorek servera HTTP bikar bînin nodejs-stub-server ji Maxim Ignatenko

Nodejs-stub-server rpm tune. Ev e ji bo wê rpm çêbikin. rpm dê bi kar were berhev kirin Fedora Copr

Depoya antonpatsev/nodejs-stub-server zêde bikin

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Nodejs-stub-server, pîvana Apache û multiplexera termînalê ekranê li ser hemî pêşkêşkeran saz bikin

yum -y install stub_http_server screen mc httpd-tools screen

Min dema bersivê ya stub_http_server di pelê /var/lib/stub_http_server/stub_http_server.js de rast kir da ku bêtir têketin hebin.

var max_sleep = 10;

Ka em stub_http_server dest pê bikin.

systemctl start stub_http_server
systemctl enable stub_http_server

Sazkirina Clickhouse li ser server 3

ClickHouse koma rêwerzên SSE 4.2 bikar tîne, ji ber vê yekê heya ku wekî din neyê destnîşan kirin, piştgirîya wê di pêvajoya ku hatî bikar anîn de dibe pêdivîyek pergalê. Li vir emrê ye ku hûn kontrol bikin ka pêvajoya heyî SSE 4.2 piştgirî dike:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Pêşî hûn hewce ne ku depoya fermî ve girêdin:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

Ji bo sazkirina pakêtan hûn hewce ne ku emrên jêrîn bimeşînin:

sudo yum install -y clickhouse-server clickhouse-client

Destûrê bide clickhouse-server ku di pelê de /etc/clickhouse-server/config.xml guh bide qerta torê

<listen_host>0.0.0.0</listen_host>

Guhertina asta têketinê ji şopê berbi debugê

debug

Mîhengên compression standard:

min_compress_block_size  65536
max_compress_block_size  1048576

Ji bo çalakkirina Zstd compression, hate şîret kirin ku meriv dest nede mîhengê, lê li şûna DDL bikar bîne.

Min nekarî bibînim ka meriv çawa bi DDL-ê di Google-ê de bihevra zstd bikar tîne. Ji ber vê yekê min ew wekî xwe hişt.

Hevalên ku di Clickhouse de compression zstd bikar tînin, ji kerema xwe rêwerzan parve bikin.

Ji bo destpêkirina serverê wekî daemon, bixebitin:

service clickhouse-server start

Naha em biçin sazkirina Clickhouse

Biçe Clickhouse

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 - IP-ya servera ku Clickhouse lê hatî saz kirin.

Werin em databasek vektorê biafirînin

CREATE DATABASE vector;

Ka em kontrol bikin ku databas heye.

show databases;

Tabloyek vector.logs çêbikin.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

Em kontrol dikin ku tablo hatine çêkirin. Ka em dest pê bikin clickhouse-client û daxwazek bike.

Ka em biçin databasa vektorê.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Ka em li tabloyan binêrin.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Sazkirina elasticsearch-ê li ser servera 4-an da ku heman daneyan ji Elasticsearch re ji bo berhevdana bi Clickhouse re bişîne

Mifteyek rpm ya gelemperî zêde bikin

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Ka em 2 repo ava bikin:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Elasticsearch û kibana saz bikin

yum install -y kibana elasticsearch

Ji ber ku ew ê di 1 kopî de be, hûn hewce ne ku jêrîn li pelê /etc/elasticsearch/elasticsearch.yml zêde bikin:

discovery.type: single-node

Ji ber vê yekê ew vektor dikare ji serverek din re daneyan bişîne elasticsearch, werin em network.host biguherînin.

network.host: 0.0.0.0

Ji bo girêdana bi kibana re, di pelê /etc/kibana/kibana.yml de parametreya server.host biguherînin.

server.host: "0.0.0.0"

Kevin û elasticsearch-ê di destpêka otomatîkê de vedihewîne

systemctl enable elasticsearch
systemctl start elasticsearch

û kibana

systemctl enable kibana
systemctl start kibana

Veavakirina Elasticsearch-ê ji bo moda yek-girêk 1 şirîn, 0 replica. Bi îhtîmaleke mezin hûn ê komek ji hejmareke mezin pêşkêşkeran hebin û hûn ne hewce ne ku vê yekê bikin.

Ji bo navnîşên pêşerojê, şablonê xwerû nûve bikin:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' 

mîhengê Vektor wekî şûna Logstash li ser server 2

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Werin em Vector wekî şûna Logstash saz bikin. Guhertina pelê /etc/vector/vector.toml

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

Hûn dikarin beşa transforms.nginx_parse_add_defaults eyar bikin.

ji ber ku Vyacheslav Rakhinsky van mîhengan ji bo CDNek piçûk bikar tîne û di jorîn_* de dikare çend nirx hebin

Bo nimûne:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Ger ev ne rewşa we ye, wê hingê ev beş dikare were hêsan kirin

Ka em ji bo systemd /etc/systemd/system/vector.service mîhengên karûbarê biafirînin

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Piştî çêkirina tabloyan, hûn dikarin Vector bimeşînin

systemctl enable vector
systemctl start vector

Têketinên vektorî dikarin bi vî rengî werin dîtin:

journalctl -f -u vector

Divê navnîşên bi vî rengî di qeydan de hebin

INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.

Li ser xerîdar (pêşkêşkara malperê) - pêşkêşkara 1

Li ser servera bi nginx, hûn hewce ne ku ipv6 neçalak bikin, ji ber ku tabloya têketinê li clickhouse zeviyê bikar tîne upstream_addr IPv4, ji ber ku ez ipv6 di hundurê torê de bikar naynim. Ger ipv6 neyê girtin, dê xeletî hebin:

DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)

Dibe ku xwendevanan, piştgiriya ipv6 zêde bikin.

Pelê /etc/sysctl.d/98-disable-ipv6.conf biafirîne

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Sepandina mîhengan

sysctl --system

Ka em nginx saz bikin.

Pelê depoya nginx /etc/yum.repos.d/nginx.repo zêde kir

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Pakêta nginx saz bikin

yum install -y nginx

Pêşî, pêdivî ye ku em di pelê /etc/nginx/nginx.conf de formata têketinê ya Nginx-ê mîheng bikin.

user  nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Ji bo ku hûn veavakirina xweya heyî neşikînin, Nginx dihêle hûn çend rêwerzên gihîştina_logê hebin

access_log  /var/log/nginx/access.log  main;            # Стандартный лог
access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

Ji bîr nekin ku ji bo têketinên nû qaîdeyek lê zêde bikin (heke pelê têketinê bi .log neqede)

default.conf ji /etc/nginx/conf.d/ rake

rm -f /etc/nginx/conf.d/default.conf

Mêvandarê virtual lê zêde bike /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Mêvandarê virtual lê zêde bike /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Mêvandarê virtual lê zêde bike /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Mêvandarê virtual lê zêde bike /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Mêvanên virtual (172.26.10.106 ip ya servera ku nginx lê hatî saz kirin) li hemî serveran li pelê /etc/hosts zêde bikin:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Û heke her tişt amade ye wê hingê

nginx -t 
systemctl restart nginx

Niha em bi xwe saz bikin Vektor

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Ka em ji bo systemd /etc/systemd/system/vector.service pelek mîhengan çêbikin

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Û di veavakirina /etc/vector/vector.toml de veguhertina Filebeat mîheng bikin. Navnîşana IP-ê 172.26.10.108 navnîşana IP-ya servera têketinê ye (Vector-Server)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Ji bîr nekin ku vektora bikarhêner li koma guncaw zêde bikin da ku ew bikaribe pelên têketinê bixwîne. Mînakî, nginx di centos bi mafên koma adm têketinan diafirîne.

usermod -a -G adm vector

Werin em dest bi karûbarê vektorê bikin

systemctl enable vector
systemctl start vector

Têketinên vektorî dikarin bi vî rengî werin dîtin:

journalctl -f -u vector

Divê navnîşek bi vî rengî di qeydan de hebe

INFO vector::topology::builder: Healthcheck: Passed.

Testkirina Stresê

Em ceribandinê bi karanîna pîvana Apache pêk tînin.

Pakêta httpd-tools li ser hemî pêşkêşkeran hate saz kirin

Em bi karanîna pîvana Apache ji 4 serverên cihêreng ên li ser ekranê dest bi ceribandinê dikin. Pêşîn, em multiplekserê termînalê ekranê didin destpêkirin, û dûv re jî em bi karanîna pîvana Apache dest bi ceribandinê dikin. Meriv çawa bi ekranê re dixebite ku hûn dikarin tê de bibînin gotara.

Ji servera 1st

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Ji servera 2st

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Ji servera 3st

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Ji servera 4st

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Ka em daneyên li Clickhouse kontrol bikin

Biçe Clickhouse

clickhouse-client -h 172.26.10.109 -m

Çêkirina pirsek SQL

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴──────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Mezinahiya tabloyên li Clickhouse bibînin

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Ka em fêr bibin ka çiqas têketin li Clickhouse girt.

Mezinahiya tabloya têketin 857.19 MB e.

Mezinahiya heman daneyê di navnîşa Elasticsearch de 4,5 GB ye.

Heke hûn di vektorê de di pîvanan de daneyan diyar nekin, Clickhouse 4500/857.19 = 5.24 carî ji Elasticsearch kêmtir digire.

Di vektorê de, qada kompresyonê ji hêla xwerû ve tê bikar anîn.

chat Telegram ji aliyê Clickhouse
chat Telegram ji aliyê Elasticsearch
chat Telegram ji hêla "Berhevkirin û analîzkirina pergalê peyam"

Source: www.habr.com