Fifiranṣẹ awọn iwe Nginx json ni lilo Vector si Clickhouse ati Elasticsearch

Vector, ṣe apẹrẹ lati gba, yipada ati firanṣẹ data log, awọn metiriki ati awọn iṣẹlẹ.

→ Github

Ni kikọ ni ede Rust, o jẹ ijuwe nipasẹ iṣẹ giga ati agbara Ramu kekere ni akawe si awọn afọwọṣe rẹ. Ni afikun, akiyesi pupọ ni a san si awọn iṣẹ ti o ni ibatan si atunṣe, ni pataki, agbara lati ṣafipamọ awọn iṣẹlẹ ti a ko firanṣẹ si ifipamọ lori disiki ati yiyi awọn faili.

Ni ayaworan, Vector jẹ olulana iṣẹlẹ ti o gba awọn ifiranṣẹ lati ọkan tabi diẹ sii awọn orisun, optionally nbere lori awọn wọnyi awọn ifiranṣẹ awọn iyipada, ati fifiranṣẹ wọn si ọkan tabi diẹ ẹ sii drains.

Vector jẹ aropo fun filebeat ati logstash, o le ṣe ni awọn ipa mejeeji (gba ati firanṣẹ awọn akọọlẹ), awọn alaye diẹ sii lori wọn Aaye.

Ti o ba wa ni Logstash pq ti wa ni itumọ bi titẹ sii → àlẹmọ → iṣelọpọ lẹhinna ni Vector o jẹ awọn orisun → awọn iyipada → rii

Awọn apẹẹrẹ le wa ninu iwe-ipamọ.

Ilana yii jẹ itọnisọna atunṣe lati Vyacheslav Rakhinsky. Awọn itọnisọna atilẹba ni sisẹ geoip ninu. Nigbati o ba ṣe idanwo geoip lati inu nẹtiwọọki inu, fekito fun aṣiṣe kan.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30

Ti ẹnikẹni ba nilo lati ṣe ilana geoip, lẹhinna tọka si awọn itọnisọna atilẹba lati Vyacheslav Rakhinsky.

A yoo tunto apapo Nginx (Awọn igbasilẹ Wiwọle) → Vector (Onibara | Filebeat) → Vector (Olupin | Logstash) → lọtọ ni Clickhouse ati lọtọ ni Elasticsearch. A yoo fi sori ẹrọ 4 olupin. Botilẹjẹpe o le fori rẹ pẹlu awọn olupin 3.

Ilana naa jẹ nkan bi eyi.

Pa Selinux kuro lori gbogbo awọn olupin rẹ

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot

A fi emulator olupin HTTP sori ẹrọ + awọn ohun elo lori gbogbo awọn olupin

Gẹgẹbi emulator olupin HTTP a yoo lo nodejs-stub-olupin lati Maxim Ignatenko

Nodejs-stub-server ko ni rpm kan. o ti wa ni ṣẹda rpm fun u. rpm yoo ṣe akopọ nipa lilo Fedora Copr

Fi antonpatsev/nodejs-stub-server ibi ipamọ sii

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Fi nodejs-stub-server sori ẹrọ, aami Apache ati multiplexer ebute iboju lori gbogbo awọn olupin

yum -y install stub_http_server screen mc httpd-tools screen

Mo ṣe atunṣe akoko idahun stub_http_server ninu faili /var/lib/stub_http_server/stub_http_server.js ki awọn akọọlẹ diẹ sii wa.

var max_sleep = 10;

Jẹ ki a ṣe ifilọlẹ stub_http_server.

systemctl start stub_http_server
systemctl enable stub_http_server

Clickhouse fifi sori lori olupin 3

ClickHouse nlo eto itọnisọna SSE 4.2, nitorinaa ayafi ti bibẹẹkọ pato, atilẹyin fun rẹ ninu ero isise ti a lo di ibeere eto afikun. Eyi ni aṣẹ lati ṣayẹwo boya ero isise lọwọlọwọ ṣe atilẹyin SSE 4.2:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Ni akọkọ o nilo lati sopọ ibi ipamọ osise:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

Lati fi sori ẹrọ awọn idii o nilo lati ṣiṣe awọn aṣẹ wọnyi:

sudo yum install -y clickhouse-server clickhouse-client

Gba aaye-olupin laaye lati tẹtisi kaadi nẹtiwọki ninu faili /etc/clickhouse-server/config.xml

<listen_host>0.0.0.0</listen_host>

Yiyipada ipele gedu lati itọpa si yokokoro

yokokoro

Awọn eto funmorawon to peye:

min_compress_block_size  65536
max_compress_block_size  1048576

Lati mu funmorawon Zstd ṣiṣẹ, o gba ọ niyanju lati ma fi ọwọ kan atunto, ṣugbọn dipo lati lo DDL.

Emi ko le rii bii o ṣe le lo funmorawon zstd nipasẹ DDL ni Google. Nitorina ni mo fi silẹ bi o ti jẹ.

Awọn ẹlẹgbẹ ti o lo zstd funmorawon ni Clickhouse, jọwọ pin awọn ilana.

Lati bẹrẹ olupin bi daemon, ṣiṣe:

service clickhouse-server start

Bayi jẹ ki a lọ si eto Clickhouse

Lọ si Clickhouse

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 - IP ti olupin ibi ti Clickhouse ti fi sori ẹrọ.

Jẹ ká ṣẹda a fekito database

CREATE DATABASE vector;

Jẹ ki a ṣayẹwo pe database wa.

show databases;

Ṣẹda a vector.logs tabili.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

A ṣayẹwo pe a ti ṣẹda awọn tabili. Jẹ ki a ṣe ifilọlẹ clickhouse-client ki o si ṣe kan ìbéèrè.

Jẹ ká lọ si awọn fekito database.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Jẹ ká wo ni awọn tabili.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Fifi elasticsearch sori olupin 4th lati firanṣẹ data kanna si Elasticsearch fun lafiwe pẹlu Clickhouse

Ṣafikun bọtini rpm ti gbogbo eniyan

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Jẹ ki a ṣẹda repo 2:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Fi elasticsearch sori ẹrọ ati kibana

yum install -y kibana elasticsearch

Niwọn igba ti yoo wa ni ẹda 1, o nilo lati ṣafikun atẹle naa si faili /etc/elasticsearch/elasticsearch.yml:

discovery.type: single-node

Ki vector le fi data ranṣẹ si elasticsearch lati olupin miiran, jẹ ki a yipada network.host.

network.host: 0.0.0.0

Lati sopọ si kibana, yi paramita olupin olupin.host pada ninu faili /etc/kibana/kibana.yml

server.host: "0.0.0.0"

Atijọ ati pẹlu wiwa elastick ni autostart

systemctl enable elasticsearch
systemctl start elasticsearch

ati kibana

systemctl enable kibana
systemctl start kibana

Ṣiṣeto Elasticsearch fun ipo-ipin kan 1 shard, 0 ajọra. O ṣeese julọ iwọ yoo ni iṣupọ ti nọmba nla ti awọn olupin ati pe iwọ ko nilo lati ṣe eyi.

Fun awọn atọka ọjọ iwaju, ṣe imudojuiwọn awoṣe aiyipada:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' 

eto Vector bi aropo fun Logstash lori olupin 2

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Jẹ ki a ṣeto Vector bi aropo fun Logstash. Ṣatunkọ faili /etc/vector/vector.toml

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

O le ṣatunṣe apakan transforms.nginx_parse_add_defaults.

Bi Vyacheslav Rakhinsky nlo awọn atunto wọnyi fun CDN kekere ati pe awọn iye pupọ le wa ni oke_*

Fun apere:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Ti eyi kii ṣe ipo rẹ, lẹhinna apakan yii le jẹ irọrun

Jẹ ki a ṣẹda awọn eto iṣẹ fun systemd /etc/systemd/system/vector.service

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Lẹhin ṣiṣẹda awọn tabili, o le ṣiṣe Vector

systemctl enable vector
systemctl start vector

Awọn akọọlẹ Vector le ṣee wo bi eleyi:

journalctl -f -u vector

Iru awọn titẹ sii yẹ ki o wa ninu awọn akọọlẹ

INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.

Lori alabara (olupin wẹẹbu) - olupin 1st

Lori olupin pẹlu nginx, o nilo lati mu ipv6 kuro, niwọn igba ti tabili awọn akọọlẹ ni ile tẹ nlo aaye naa upstream_addr IPv4, niwon Emi ko lo ipv6 inu nẹtiwọki. Ti a ko ba pa ipv6, awọn aṣiṣe yoo wa:

DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)

Boya awọn oluka, ṣafikun atilẹyin ipv6.

Ṣẹda faili /etc/sysctl.d/98-disable-ipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Lilo awọn eto

sysctl --system

Jẹ ki a fi nginx sori ẹrọ.

Ṣafikun faili ibi ipamọ nginx /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Fi package nginx sori ẹrọ

yum install -y nginx

Ni akọkọ, a nilo lati tunto ọna kika log ni Nginx ninu faili /etc/nginx/nginx.conf

user  nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Ni ibere ki o má ba fọ iṣeto rẹ lọwọlọwọ, Nginx gba ọ laaye lati ni ọpọlọpọ awọn itọnisọna wiwọle_log

access_log  /var/log/nginx/access.log  main;            # Стандартный лог
access_log  /var/log/nginx/access.json.log vector;      # Новый лог в формате json

Maṣe gbagbe lati ṣafikun ofin kan si logrotate fun awọn akọọlẹ tuntun (ti faili log ko ba pari pẹlu .log)

Yọ default.conf lati /etc/nginx/conf.d/

rm -f /etc/nginx/conf.d/default.conf

Ṣafikun alejo gbigba foju /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Ṣafikun alejo gbigba foju /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Ṣafikun alejo gbigba foju /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Ṣafikun alejo gbigba foju /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Ṣafikun awọn ogun foju (172.26.10.106 ip ti olupin nibiti nginx ti fi sii) si gbogbo awọn olupin si faili /etc/hosts:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Ati pe ti ohun gbogbo ba ṣetan lẹhinna

nginx -t 
systemctl restart nginx

Bayi jẹ ki a fi sori ẹrọ funrararẹ Vector

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Jẹ ki a ṣẹda faili eto fun systemd /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Ati tunto rirọpo Filebeat ni atunto /etc/vector/vector.toml. Adirẹsi IP 172.26.10.108 jẹ adiresi IP ti olupin log (Vector-Server)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Maṣe gbagbe lati ṣafikun olumulo fekito si ẹgbẹ ti o nilo ki o le ka awọn faili log. Fun apẹẹrẹ, nginx ni centos ṣẹda awọn akọọlẹ pẹlu awọn ẹtọ ẹgbẹ adm.

usermod -a -G adm vector

Jẹ ká bẹrẹ iṣẹ fekito

systemctl enable vector
systemctl start vector

Awọn akọọlẹ Vector le ṣee wo bi eleyi:

journalctl -f -u vector

O yẹ ki o jẹ titẹ sii bi eyi ninu awọn akọọlẹ

INFO vector::topology::builder: Healthcheck: Passed.

Idanwo Wahala

A ṣe idanwo ni lilo ala Apache.

Awọn ohun elo httpd-irinṣẹ ti fi sori ẹrọ lori gbogbo awọn olupin

A bẹrẹ idanwo ni lilo ala Apache lati awọn olupin oriṣiriṣi mẹrin ni iboju. Ni akọkọ, a ṣe ifilọlẹ multixer ebute iboju, lẹhinna a bẹrẹ idanwo ni lilo ala Apache. Bii o ṣe le ṣiṣẹ pẹlu iboju o le wa ninu article.

Lati olupin 1st

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Lati olupin 2st

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Lati olupin 3st

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Lati olupin 4st

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Jẹ ki a ṣayẹwo data ni Clickhouse

Lọ si Clickhouse

clickhouse-client -h 172.26.10.109 -m

Ṣiṣe ibeere SQL kan

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴──────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Wa iwọn awọn tabili ni Clickhouse

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Jẹ ki a wa iye awọn akọọlẹ ti o gba ni Clickhouse.

Iwọn tabili awọn akọọlẹ jẹ 857.19 MB.

Iwọn data kanna ni atọka ni Elasticsearch jẹ 4,5GB.

Ti o ko ba pato data ninu fekito ninu awọn paramita, Clickhouse gba 4500/857.19 = 5.24 igba kere ju ni Elasticsearch.

Ni fekito, aaye funmorawon ti lo nipasẹ aiyipada.

Telegram iwiregbe nipasẹ Tẹ ile
Telegram iwiregbe nipasẹ Elasticsearch
Iwiregbe Telegram nipasẹ"Gbigba ati igbekale ti eto awọn ifiranṣẹ"

orisun: www.habr.com