🥇Send Nginx json logs med Vector til Clickhouse og Elasticsearch

Vector, beregnet til at samle, konvertere og sende logdata, målinger og begivenheder.

Skriver den i Rust, har den høj ydeevne og lavt hukommelsesforbrug sammenlignet med tilsvarende værktøjer. Desuden er der lagt særlig vægt på funktioner relateret til korrekthed, specielt muligheden for at gemme uafsendte begivenheder i en buffer på disken og rotation af filer.

Arkitektonisk set er Vector en begivenhedsrouter, der modtager meddelelser fra en eller flere kilder, valgfrit anvender omformninger, og sender dem til en eller flere udgange..

Vector er en erstatning for filebeat og logstash; den kan fungere i begge roller (modtage og sende logs), mere detaljeret om dem. websitet.

Hvis en kæde i Logstash konstrueres som input → filter → output, så er det i Vector. sources → transforms → sinks

Eksempler kan ses i dokumentationen.

Denne vejledning er en bearbejdet version af Vyacheslav Rakhinskiys. I den oprindelige vejledning er der geoip-behandling. Jeg fik en fejl under testning af geoip fra det interne netværk.

Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Felt eksisterede ikke field=«geoip.country_name» rate_limit_secs=30

Hvis nogen har brug for at behandle geoip, så kontakt den oprindelige vejledning fra Vyacheslav Rakhinskiys.

Vi vil konfigurere kombinationen Nginx (Access logs) → Vector (Client | Filebeat) → Vector (Server | Logstash) → separat i Clickhouse og separat Elasticsearch. Vi opretter 4 servere. Selvom vi kan klare os med 3 servere.

Skemaet ser omtrent sådan ud.

Vi deaktiverer Selinux på alle jeres servere.

sed -i 's/^SELINUX=.* /SELINUX=disabled/g' /etc/selinux/config
reboot

Vi installerer en HTTP server emulator + værktøjer på alle servere.

Som HTTP server emulator vil vi bruge nodejs-stub-server fra Maxim Ignatenko

Nodejs-stub-server har ikke nogen rpm. Her opretter vi en rpm for det. RPM'en vil blive bygget ved hjælp af Fedora Copr

Tilføj lageret antonpatsev/nodejs-stub-server.

yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-server

Installerer nodejs-stub-server, Apache-benchmark og terminalmultiplexer screen på alle servere.

yum -y install stub_http_server screen mc httpd-tools screen

Jeg har rettet i filen /var/lib/stub_http_server/stub_http_server.js svartiden for stub_http_server, så der er flere logs.

var max_sleep = 10;

Lad os starte stub_http_server.

systemctl start stub_http_server
systemctl enable stub_http_server

Installation af Clickhouse. på server 3.

ClickHouse bruger et sæt instruktioner SSE 4.2, så hvis andet ikke er angivet, bliver dets støtte i den anvendte processor et ekstra systemkrav. Her er kommandoen til at kontrollere, om den nuværende processor understøtter SSE 4.2:

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

Først skal vi tilslutte det officielle lager:

sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

For at installere pakkerne skal følgende kommandoer udføres:

sudo yum install -y clickhouse-server clickhouse-client

Vi tillader clickhouse-server at lytte på netværkskortet i filen /etc/clickhouse-server/config.xml.

0.0.0.0

Vi ændrer logningsniveauet fra trace til debug.

debug

Standard komprimeringsindstillinger:

min_compress_block_size  65536
max_compress_block_size  1048576

For activating Zstd compression, it is recommended to leave the config untouched and instead apply DDL.

I couldn't find how to apply zstd compression via DDL on Google. So I left it as is.

Colleagues, anyone using zstd compression in Clickhouse — please share instructions.

To start the server as a daemon, execute:

service clickhouse-server start

Now, let's proceed to configuring Clickhouse.

clickhouse-client -h 172.26.10.109 -m

172.26.10.109 — the IP of the server where Clickhouse is installed.

Let's create the DB vector.

CREATE DATABASE vector;

Let's check if the database exists.

show databases;

Creating the table vector.logs.

/* Это таблица где хранятся логи как есть */

CREATE TABLE vector.logs
(
    `node_name` String,
    `timestamp` DateTime,
    `server_name` String,
    `user_id` String,
    `request_full` String,
    `request_user_agent` String,
    `request_http_host` String,
    `request_uri` String,
    `request_scheme` String,
    `request_method` String,
    `request_length` UInt64,
    `request_time` Float32,
    `request_referrer` String,
    `response_status` UInt16,
    `response_body_bytes_sent` UInt64,
    `response_content_type` String,
    `remote_addr` IPv4,
    `remote_port` UInt32,
    `remote_user` String,
    `upstream_addr` IPv4,
    `upstream_port` UInt32,
    `upstream_bytes_received` UInt64,
    `upstream_bytes_sent` UInt64,
    `upstream_cache_status` String,
    `upstream_connect_time` Float32,
    `upstream_header_time` Float32,
    `upstream_response_length` UInt64,
    `upstream_response_time` Float32,
    `upstream_status` UInt16,
    `upstream_content_type` String,
    INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;

Checking if the tables were created. Running clickhouse-client and making a query.

Let's move to the database vector.

use vector;

Ok.

0 rows in set. Elapsed: 0.001 sec.

Looking at the tables.

show tables;

┌─name────────────────┐
│ logs                │
└─────────────────────┘

Installing Elasticsearch on the 4th server to send the same data to Elasticsearch for comparison with Clickhouse.

Let's add the public rpm key.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Let's create 2 repos:

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

/etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

We will install Elasticsearch and Kibana.

yum install -y kibana elasticsearch

Since it will be in a single instance, we need to add to the file /etc/elasticsearch/elasticsearch.yml:

discovery.type: single-node

To enable vector to send data to Elasticsearch from another server, we will change the network.host.

network.host: 0.0.0.0

To connect to Kibana, change the server.host parameter in the file /etc/kibana/kibana.yml.

server.host: "0.0.0.0"

We start and enable Elasticsearch to autostart.

systemctl enable elasticsearch
systemctl start elasticsearch

and Kibana.

systemctl enable kibana
systemctl start kibana

Configuring Elasticsearch for single-node mode with 1 shard, 0 replica. Most likely, you will have a cluster with a large number of servers, and you don't need to do this.

For future indexes, we update the default template:

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}'

Installation Vector as a replacement for Logstash on the 2nd server.

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screen

Let's configure Vector as a replacement for Logstash. Edit the file /etc/vector/vector.toml.

# /etc/vector/vector.toml

data_dir = "/var/lib/vector"

[sources.nginx_input_vector]
  # General
  type                          = "vector"
  address                       = "0.0.0.0:9876"
  shutdown_timeout_secs         = 30

[transforms.nginx_parse_json]
  inputs                        = [ "nginx_input_vector" ]
  type                          = "json_parser"

[transforms.nginx_parse_add_defaults]
  inputs                        = [ "nginx_parse_json" ]
  type                          = "lua"
  version                       = "2"

  hooks.process = """
  function (event, emit)

    function split_first(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[1];
    end

    function split_last(s, delimiter)
      result = {};
      for match in (s..delimiter):gmatch("(.-)"..delimiter) do
          table.insert(result, match);
      end
      return result[#result];
    end

    event.log.upstream_addr             = split_first(split_last(event.log.upstream_addr, ', '), ':')
    event.log.upstream_bytes_received   = split_last(event.log.upstream_bytes_received, ', ')
    event.log.upstream_bytes_sent       = split_last(event.log.upstream_bytes_sent, ', ')
    event.log.upstream_connect_time     = split_last(event.log.upstream_connect_time, ', ')
    event.log.upstream_header_time      = split_last(event.log.upstream_header_time, ', ')
    event.log.upstream_response_length  = split_last(event.log.upstream_response_length, ', ')
    event.log.upstream_response_time    = split_last(event.log.upstream_response_time, ', ')
    event.log.upstream_status           = split_last(event.log.upstream_status, ', ')

    if event.log.upstream_addr == "" then
        event.log.upstream_addr = "127.0.0.1"
    end

    if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
        event.log.upstream_bytes_received = "0"
    end

    if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
        event.log.upstream_bytes_sent = "0"
    end

    if event.log.upstream_cache_status == "" then
        event.log.upstream_cache_status = "DISABLED"
    end

    if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
        event.log.upstream_connect_time = "0"
    end

    if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
        event.log.upstream_header_time = "0"
    end

    if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
        event.log.upstream_response_length = "0"
    end

    if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
        event.log.upstream_response_time = "0"
    end

    if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
        event.log.upstream_status = "0"
    end

    emit(event)

  end
  """

[transforms.nginx_parse_remove_fields]
    inputs                              = [ "nginx_parse_add_defaults" ]
    type                                = "remove_fields"
    fields                              = ["data", "file", "host", "source_type"]

[transforms.nginx_parse_coercer]

    type                                = "coercer"
    inputs                              = ["nginx_parse_remove_fields"]

    types.request_length = "int"
    types.request_time = "float"

    types.response_status = "int"
    types.response_body_bytes_sent = "int"

    types.remote_port = "int"

    types.upstream_bytes_received = "int"
    types.upstream_bytes_send = "int"
    types.upstream_connect_time = "float"
    types.upstream_header_time = "float"
    types.upstream_response_length = "int"
    types.upstream_response_time = "float"
    types.upstream_status = "int"

    types.timestamp = "timestamp"

[sinks.nginx_output_clickhouse]
    inputs   = ["nginx_parse_coercer"]
    type     = "clickhouse"

    database = "vector"
    healthcheck = true
    host = "http://172.26.10.109:8123" #  Адрес Clickhouse
    table = "logs"

    encoding.timestamp_format = "unix"

    buffer.type = "disk"
    buffer.max_size = 104900000
    buffer.when_full = "block"

    request.in_flight_limit = 20

[sinks.elasticsearch]
    type = "elasticsearch"
    inputs   = ["nginx_parse_coercer"]
    compression = "none"
    healthcheck = true
    # 172.26.10.116 - сервер где установен elasticsearch
    host = "http://172.26.10.116:9200" 
    index = "vector-%Y-%m-%d"

You can adjust the section transforms.nginx_parse_add_defaults.

Da Vyacheslav Rakhinsky uses these configs for a small CDN, and there in upstream_* multiple values may come in.

For eksempel:

"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"

Hvis dette ikke er din situation, kan denne sektion forenkles

Lad os oprette serviceindstillinger til systemd /etc/systemd/system/vector.service

# /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Efter oprettelse af tabellerne kan Vector startes

systemctl enable vector
systemctl start vector

Logfilerne fra Vector kan ses sådan

journalctl -f -u vector

Logfilerne skal have sådanne poster

INFO vector::topology::builder: Healthcheck: Bestået.
INFO vector::topology::builder: Healthcheck: Bestået.

På klienten (Webserver) — 1. server

På serveren med nginx skal IPv6 deaktiveres, da feltet upstream_addr IPv4, da jeg ikke bruger IPv6 i netværket. Hvis IPv6 ikke deaktiveres, vil der opstå fejl:

DB::Exception: Ugyldig IPv4-værdi.: (ved læsning af værdien af nøgle upstream_addr)

Det er muligt for læserne at tilføje støtte til IPv6.

Vi opretter filen /etc/sysctl.d/98-disable-ipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Anvend indstillingerne

sysctl --system

Vi installerer nginx.

Jeg har tilføjet nginx-repositories filen /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stabilt repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Vi installerer nginx-pakken

yum install -y nginx

For at starte skal vi konfigurere logformatet i Nginx i filen /etc/nginx/nginx.conf

bruger  nginx;
# du skal indstille arbejdstprocesser baseret på dine CPU-kerner, nginx får ikke noget ud af at indstille mere end det
worker_processes auto; #nogle seneste versioner beregner det automatisk

# antal filbeskrivelser, der bruges til nginx
# grænsen for de maksimale FDs på serveren er normalt indstillet af OS.
# hvis du ikke indstiller FDs, vil OS-indstillingerne blive brugt, hvilket som standard er 2000
worker_rlimit_nofile 100000;

error_log  	/var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# giver konfigurationsfilens kontekst, hvori direktiverne, der påvirker forbindelsebehandlingen, er specificeret.
events {
    # bestemmer, hvor mange klienter der serveres pr. arbejder
    # maksimale klienter = worker_connections * worker_processes
    # maksimale klienter er også begrænset af antallet af socket-forbindelser, der er tilgængelige på systemet (~64k)
    worker_connections 4000;

    # optimeret til at servere mange klienter med hver tråd, essentiel for linux -- til testmiljø
    use epoll;

    # accepterer så mange forbindelser som muligt, kan oversvømme arbejdstilslutninger, hvis indstillet for lavt -- til testmiljø
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format vector escape=json
    '{'
        '"node_name":"nginx-vector",'
        '"timestamp":"$time_iso8601",'
        '"server_name":"$server_name",'
        '"request_full": "$request",'
        '"request_user_agent":"$http_user_agent",'
        '"request_http_host":"$http_host",'
        '"request_uri":"$request_uri",'
        '"request_scheme": "$scheme",'
        '"request_method":"$request_method",'
        '"request_length":"$request_length",'
        '"request_time": "$request_time",'
        '"request_referrer":"$http_referer",'
        '"response_status": "$status",'
        '"response_body_bytes_sent":"$body_bytes_sent",'
        '"response_content_type":"$sent_http_content_type",'
        '"remote_addr": "$remote_addr",'
        '"remote_port": "$remote_port",'
        '"remote_user": "$remote_user",'
        '"upstream_addr": "$upstream_addr",'
        '"upstream_bytes_received": "$upstream_bytes_received",'
        '"upstream_bytes_sent": "$upstream_bytes_sent",'
        '"upstream_cache_status":"$upstream_cache_status",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_status": "$upstream_status",'
        '"upstream_content_type":"$upstream_http_content_type"'
    '}';

    access_log  /var/log/nginx/access.log  main;
    access_log  /var/log/nginx/access.json.log vector;      # Ny log i json-format

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

For ikke at ødelægge din nuværende konfiguration, tillader Nginx at have flere directives access_log

access_log  /var/log/nginx/access.log  main;            # Standard log
access_log  /var/log/nginx/access.json.log vector;      # Ny log i json-format

Glem ikke at tilføje en regel til logrotate for de nye logs (hvis logfilen ikke ender med .log)

Fjern default.conf fra /etc/nginx/conf.d/

rm -f /etc/nginx/conf.d/default.conf

Tilføj en virtuel vært /etc/nginx/conf.d/vhost1.conf

server {
    listen 80;
    server_name vhost1;
    location / {
        proxy_pass http://172.26.10.106:8080;
    }
}

Tilføj en virtuel vært /etc/nginx/conf.d/vhost2.conf

server {
    listen 80;
    server_name vhost2;
    location / {
        proxy_pass http://172.26.10.108:8080;
    }
}

Tilføj en virtuel vært /etc/nginx/conf.d/vhost3.conf

server {
    listen 80;
    server_name vhost3;
    location / {
        proxy_pass http://172.26.10.109:8080;
    }
}

Tilføj en virtuel vært /etc/nginx/conf.d/vhost4.conf

server {
    listen 80;
    server_name vhost4;
    location / {
        proxy_pass http://172.26.10.116:8080;
    }
}

Tilføj virtuelle værter i filen /etc/hosts (172.26.10.106 er IP-adressen på serveren hvor Nginx er installeret) til alle servere:

172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4

Og hvis alt er klart, så

nginx -t 
systemctl restart nginx

Nu installerer vi selve Vector

yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm

Opret en konfigurationsfil til systemd /etc/systemd/system/vector.service

[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector

[Install]
WantedBy=multi-user.target

Og konfigurerer erstatning af Filebeat i konfigurationsfilen /etc/vector/vector.toml. IP-adressen 172.26.10.108 er IP-adressen på logserveren (Vector-Server)

data_dir = "/var/lib/vector"

[sources.nginx_file]
  type                          = "file"
  include                       = [ "/var/log/nginx/access.json.log" ]
  start_at_beginning            = false
  fingerprinting.strategy       = "device_and_inode"

[sinks.nginx_output_vector]
  type                          = "vector"
  inputs                        = [ "nginx_file" ]

  address                       = "172.26.10.108:9876"

Husk at tilføje brugeren vector til den nødvendige gruppe, så den kan læse logfilerne. For eksempel opretter nginx i CentOS logs med rettighederne til gruppen adm.

usermod -a -G adm vector

Lad os starte vector-tjenesten

systemctl enable vector
systemctl start vector

Logfilerne fra Vector kan ses sådan

journalctl -f -u vector

Logfilerne bør indeholde denne post

INFO vector::topology::builder: Healthcheck: Passed.

Belastningstest

Testen udføres ved hjælp af Apache benchmark.

Httpd-tools-pakken er installeret på alle servere

Vi udfører testning med Apache benchmark fra 4 forskellige servere i en screen. Først starter vi terminalmultiplexer screen, og derefter udfører vi testningen med Apache benchmark. Du kan finde ud af, hvordan man arbejder med screen i artikel.

Fra server 1

while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done

Fra server 2

while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done

Fra server 3

while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done

Fra server 4

while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done

Lad os kontrollere dataene i Clickhouse

clickhouse-client -h 172.26.10.109 -m

Foretag SQL-forespørgsel

SELECT * FROM vector.logs;

┌─node_name────┬───────────timestamp─┬─server_name─┬─user_id─┬─request_full───┬─request_user_agent─┬─request_http_host─┬─request_uri─┬─request_scheme─┬─request_method─┬─request_length─┬─request_time─┬─request_referrer─┬─response_status─┬─response_body_bytes_sent─┬─response_content_type─┬───remote_addr─┬─remote_port─┬─remote_user─┬─upstream_addr─┬─upstream_port─┬─upstream_bytes_received─┬─upstream_bytes_sent─┬─upstream_cache_status─┬─upstream_connect_time─┬─upstream_header_time─┬─upstream_response_length─┬─upstream_response_time─┬─upstream_status─┬─upstream_content_type─┐
│ nginx-vector │ 2020-08-07 04:32:42 │ vhost1      │         │ GET / HTTP/1.0 │ 1server            │ vhost1            │ /           │ http           │ GET            │             66 │        0.028 │                  │             404 │                       27 │                       │ 172.26.10.106 │       45886 │             │ 172.26.10.106 │             0 │                     109 │                  97 │ DISABLED              │                     0 │                0.025 │                       27 │                  0.029 │             404 │                       │
└──────────────┴─────────────────────┴─────────────┴─────────┴────────────────┴────────────────────┴───────────────────┴─────────────┴────────────────┴────────────────┴────────────────┴──────────────┴──────────────────┴─────────────────┴──────────────────────────┴───────────────────────┴───────────────┴─────────────┴─────────────┴───────────────┴───────────────┴─────────────────────────┴─────────────────────┴───────────────────────┴───────────────────────┴───────────────────────┴──────────────────────────┴────────────────────────┴─────────────────┴───────────────────────

Lad os finde størrelsen på tabellerne i Clickhouse

select concat(database, '.', table)                         as table,
       formatReadableSize(sum(bytes))                       as size,
       sum(rows)                                            as rows,
       max(modification_time)                               as latest_modification,
       sum(bytes)                                           as bytes_size,
       any(engine)                                          as engine,
       formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;

Lad os finde ud af, hvor meget plads loggene optager i Clickhouse.