Ukuthumela iilog ze-Nginx json usebenzisa iVector kwiClickhouse kunye ne-Elasticsearch

Vector, eyenzelwe ukuqokelela, ukuguqula kunye nokuthumela idatha yelog, i-metrics kunye neziganeko.

→ Github

Ukubhalwa ngolwimi lwe-Rust, lubonakaliswa ngokusebenza okuphezulu kunye nokusetyenziswa kwe-RAM ephantsi xa kuthelekiswa ne-analogues yayo. Ukongezelela, ingqwalasela eninzi ihlawulwa kwimisebenzi ehambelana nokuchaneka, ngokukodwa, ukukwazi ukugcina iziganeko ezingathunyelwanga kwi-buffer kwidiski kunye nokujikeleza iifayile.

Uyilo, iVector yirouter yesiganeko efumana imiyalezo evela kwenye okanye ngaphezulu imithombo yolwazi, ngokukhetha ukufaka phezu kwale miyalezo iinguqu, kwaye uzithumele kwenye okanye ngaphezulu iidreyini.

IVector yindawo yokubethelwa kwefayile kunye nelogstash, inokusebenza kuzo zombini iindima (ukufumana kunye nokuthumela iilogi), iinkcukacha ezingaphezulu kuzo. indawo.

Ukuba kwiLogstash ikhonkco lakhiwe njengegalelo → icebo lokucoca → imveliso ngoko kwiVektha iyiyo Kwimithombo → Utshintsho → iyatshona

Imizekelo inokufumaneka kumaxwebhu.

Lo myalelo ngumyalelo ohlaziyiweyo ovela Vyacheslav Rakhinsky. Imiyalelo yokuqala iqulathe inkqubo ye-geoip. Xa uvavanya i-geoip kwinethiwekhi yangaphakathi, i-vector inike impazamo.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30

Ukuba nabani na ufuna ukuqhubekekisa i-geoip, ngoko bhekisa kwimiyalelo yokuqala evela Vyacheslav Rakhinsky.

Siza kuqwalasela indibaniselwano yeNginx (uFikelelo kwilog) → Vector (Client | Filebeat) → Vector (Server | Logstash) → ngokwahlukileyo kwiClickhouse kwaye ngokwahlukileyo kwi-Elasticsearch. Siza kufaka iiseva ezi-4. Nangona ungayigqitha ngeeseva ezi-3.

Iskimu siyinto enje.

Khubaza iSelinux kuzo zonke iiseva zakho

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

Sifaka i-emulator ye-HTTP yeseva + eziluncedo kuzo zonke iiseva

Njengomenzi weseva yeHTTP esiya kuyisebenzisa nodejs-stub-server ukusuka UMaxim Ignatenko

I-Nodejs-stub-server ayinayo i-rpm. kuyinto yenza i-rpm kuyo. rpm iya kwakhiwa kusetyenziswa Fedora Copr

Yongeza i-antonpatsev/nodejs-stub-server repository

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Faka i-nodejs-stub-server, i-Apache benchmark kunye ne-screen multiplexer yesikrini kuzo zonke iiseva

yum -y install stub_http_server screen mc httpd-tools screen

Ndilungise ixesha lokuphendula le-stub_http_server kwifayile /var/lib/stub_http_server/stub_http_server.js ukuze kubekho iilog ezininzi.

var max_sleep = 10;

Masiqalise i-stub_http_server.

systemctl start stub_http_server
systemctl enable stub_http_server

Ufakelo lweClickhouse kwiseva 3

I-ClickHouse isebenzisa isethi yomyalelo we-SSE 4.2, ngoko ke ngaphandle kokuba kuchazwe ngenye indlela, inkxaso yayo kwiprosesa esetyenzisiweyo iba yimfuno eyongezelelweyo yenkqubo. Nanku umyalelo wokukhangela ukuba iprosesa yangoku iyayixhasa i-SSE 4.2:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Okokuqala kufuneka udibanise indawo yokugcina esemthethweni:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

Ukufakela iipakethe kufuneka wenze le miyalelo ilandelayo:

sudo yum install -y clickhouse-server clickhouse-client

Vumela i-clickhouse-server ukumamela ikhadi lomsebenzi womnatha kwifayile /etc/clickhouse-server/config.xml

<listen_host>0.0.0.0</listen_host>

Ukutshintsha umgangatho wokuloga ukusuka kumkhondo ukuya kwi-debug

ukuguqulwa

Iisetingi zoxinzelelo olusemgangathweni:

min_compress_block_size  65536
max_compress_block_size  1048576

Ukwenza ucinezelo lweZstd lusebenze, kuye kwacetyiswa ukuba ungachukumisi uqwalaselo, kodwa endaweni yoko usebenzise iDDL.

Andikwazanga ukufumana indlela yokusebenzisa i-zstd compression nge-DDL kuGoogle. Ndiye ndayishiya injalo.

Oogxa abasebenzisa i-zstd compression kwiClickhouse, nceda wabelane ngemiyalelo.

Ukuqala iseva njengedaemon, sebenzisa:

service clickhouse-server start

Ngoku masiqhubele phambili ukuseta iClickhouse

Yiya kwiClickhouse

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 — IP yomncedisi apho iClickhouse ifakwe khona.

Masenze isiseko sedatha yevektha

CREATE DATABASE vector;

Masijonge ukuba uvimba weenkcukacha ukhona.

show databases;

Yenza itafile yevector.logs.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

Sijonga ukuba iitafile zenziwe. Masiqalise clickhouse-client kwaye wenze isicelo.

Makhe siye kwisiseko sedatha yevektha.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Makhe sijonge ezitafileni.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Ukufakela i-elasticsearch kwi-server yesi-4 ukuthumela idatha efanayo kwi-Elasticsearch ukuze ithelekiswe kunye neClickhouse

Yongeza iqhosha likawonke-wonke le-rpm

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Masidale i-2 repo:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Faka elasticsearch kunye nekibana

yum install -y kibana elasticsearch

Kuba iya kuba kwikopi enye, kufuneka udibanise oku kulandelayo kwifayile /etc/elasticsearch/elasticsearch.yml:

discovery.type: single-node

Ke loo vector inokuthumela idatha kwi-elasticsearch kwenye iseva, masitshintshe network.host.

network.host: 0.0.0.0

Ukuqhagamshela kwi-kibana, tshintsha i-server.host parameter kwifayile /etc/kibana/kibana.yml

server.host: "0.0.0.0"

Indala kwaye iquka i-elasticsearch kwi-autostart

systemctl enable elasticsearch
systemctl start elasticsearch

kunye nekibana

systemctl enable kibana
systemctl start kibana

Ukuqwalasela i-Elasticsearch yemowudi ye-single-node i-1 shard, i-0 replica. Okunokwenzeka ukuba uya kuba neqela lenani elikhulu labancedisi kwaye awudingi ukwenza oku.

Kwizalathisi zexesha elizayo, hlaziya itemplate ehlala ikho:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' 

isicwangciso Vector njengokutshintsha kweLogstash kwiseva yesi-2

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Masiseke iVector njengokutshintsha kweLogstash. Ukuhlela ifayile /etc/vector/vector.toml

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

Ungalungisa icandelo transforms.nginx_parse_add_defaults.

ukususela Vyacheslav Rakhinsky isebenzisa olu qwalaselo lweCDN encinci kwaye kunokubakho amaxabiso amaninzi kumlambo_*

Umzekelo:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Ukuba oku akusiyo imeko yakho, ke eli candelo linokwenziwa lula

Masenze useto lwenkonzo ye-systemd /etc/systemd/system/vector.service

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Emva kokudala iitafile, unokuqhuba iVector

systemctl enable vector
systemctl start vector

Iilog zeVector zinokujongwa ngolu hlobo:

journalctl -f -u vector

Kufuneka kubekho amangeno anjengala kwiilog

INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.

Kumxhasi (umncedisi wewebhu) - umncedisi we-1

Kumncedisi nge nginx, kufuneka ukhubaze ipv6, kuba itafile yelog kwindlu yokucofa isebenzisa umhlaba. upstream_addr IPv4, kuba andisebenzisi ipv6 ngaphakathi kuthungelwano. Ukuba i-ipv6 ayicinywanga, kuya kubakho iimpazamo:

DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)

Mhlawumbi bafundi, yongeza inkxaso ye-ipv6.

Yenza ifayile /etc/sysctl.d/98-disable-ipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Ukusebenzisa useto

sysctl --system

Masifake nginx.

Yongezwe ifayile yenginx /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Faka ipakethe ye nginx

yum install -y nginx

Okokuqala, kufuneka siqwalasele ifomati yelog kwiNginx kwifayile /etc/nginx/nginx.conf

user  nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Ukuze ungaphuli ulungelelwaniso lwakho lwangoku, i-Nginx ikuvumela ukuba ube ne-access_log imiyalelo emininzi

access_log  /var/log/nginx/access.log  main;            # Стандартный лог
access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

Ungalibali ukongeza umthetho kwi-logrotate yelog ezintsha (ngaphandle kokuba ifayile yelog iphela nge.log)

Susa default.conf ukusuka /etc/nginx/conf.d/

rm -f /etc/nginx/conf.d/default.conf

Yongeza umamkeli obonakalayo /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Yongeza umamkeli obonakalayo /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Yongeza umamkeli obonakalayo /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Yongeza umamkeli obonakalayo /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Yongeza iinginginya ezibonakalayo (172.26.10.106 ip yomncedisi apho inginx ifakwe khona) kubo bonke abancedisi kwifayile /etc/hosts:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Kwaye ukuba yonke into ilungile ngoko

nginx -t 
systemctl restart nginx

Ngoku masiyifake ngokwethu Vector

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Masenze ifayile yesethingi ye-systemd /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Kwaye uqwalasele indawo yeFayilebeat kwi-/etc/vector/vector.toml config. Idilesi ye-IP 172.26.10.108 yidilesi ye-IP yomncedisi welog (Umncedisi weVector)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Ungalibali ukongeza i-user vector kwiqela elifanelekileyo ukuze ikwazi ukufunda iifayile zelog. Umzekelo, i-nginx kwi centos yenza iilog ngamalungelo eqela le-adm.

usermod -a -G adm vector

Masiqale inkonzo ye-vector

systemctl enable vector
systemctl start vector

Iilog zeVector zinokujongwa ngolu hlobo:

journalctl -f -u vector

Kufuneka kubekho ukungena okunje kwiilog

INFO vector::topology::builder: Healthcheck: Passed.

Uvavanyo loxinzelelo

Uvavanyo lwenziwa kusetyenziswa i-Apache benchmark.

Iphakheji ye-httpd yezixhobo ifakwe kuzo zonke iiseva

Siqala ukuvavanya usebenzisa i-Apache benchmark ukusuka kwiiseva ezi-4 ezahlukeneyo kwisikrini. Okokuqala, siqalisa i-terminal ye-multiplexer yesikrini, kwaye emva koko siqala ukuvavanya usebenzisa i-Apache benchmark. Usebenza njani ngesikrini onokufumana kuyo nqaku.

Ukusuka kwiseva yoku-1

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Ukusuka kwiseva yoku-2

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Ukusuka kwiseva yoku-3

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Ukusuka kwiseva yoku-4

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Makhe sijonge idatha kwiClickhouse

Yiya kwiClickhouse

clickhouse-client -h 172.26.10.109 -m

Ukwenza umbuzo weSQL

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴──────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Fumana ubungakanani beetafile eClickhouse

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Makhe sifumanise ukuba zingaphi iinkuni ezithathwe kwiClickhouse.

Ubungakanani betafile yelog yi-857.19 MB.

Ubungakanani bedatha efanayo kwisalathiso kwi-Elasticsearch yi-4,5GB.

Ukuba awukhankanyi idatha kwi-vector kwiiparameters, i-Clickhouse ithatha i-4500 / 857.19 = amaxesha angama-5.24 ngaphantsi kwe-Elasticsearch.

Kwi-vector, indawo yoxinzelelo isetyenziswa ngokungagqibekanga.

Incoko yeTelegram ngu indawo yokucofa
Incoko yeTelegram ngu Elasticsearch
Incoko yeTelegram ngu "Ukuqokelelwa kunye nohlalutyo lwenkqubo imiyalezo"

umthombo: www.habr.com