Nginx-log-collector, Avito-nun Clickhouse-a nginx loglarını göndərmək üçün bir vasitədir

Bu məqalədə layihə müzakirə olunacaq nginx-log-collector, nginx loglarını oxuyacaq, onları Clickhouse klasterinə göndərəcək. Adətən loglar üçün ElasticSearch istifadə olunur. Clickhouse daha az resurs tələb edir (disk sahəsi, RAM, CPU). Clickhouse məlumatları daha sürətli yazır. Clickhouse məlumatları sıxışdırır ki, diskdəki məlumatlar daha da kompakt olsun. Clickhouse-un üstünlükləri iki təqdimatdan görünür VK, on minlərlə serverdən ClickHouse-a necə məlumat daxil edir.

Loglar üzrə analitikaları görmək üçün Grafana üçün bir dashboard yaradaq.

İlginizi çəkirsə, kulisyə xoş gəlmisiniz.

Nginx-i standart şəkildə quraşdırırıq, grafana.

Clickhouse klasterini Denis Proskurin-in ansible-playbook-u ilə quraşdırırıq. Denis Proskurin.

Clickhouse-da verilənlərin və cədvəllərin yaradılması

Bu faylda nginx-log-collector üçün Clickhouse-da verilənlərin və cədvəllərin yaradılması üçün SQL sorğuları təsvir edilir.

Hər bir sorğunu Clickhouse klasterinin hər serverinə sıra ilə edirik.

Əhəmiyyətli qeydə alınma. Bu sətirdə logs_cluster-ı clickhouse_remote_servers.xml faylından "remote_servers" ilə "shard" arasında olan klaster adınızla əvəz edin.

ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())

nginx-log-collector-rpm quraşdırma və konfiqurasiya

Nginx-log-collector-in rpm-i yoxdur. Burada https://github.com/patsevanton/nginx-log-collector-rpm ona rpm yaradırıq. rpm, Fedora Copr ilə yığılacaq

nginx-log-collector-rpm rpm paketini quraşdırırıq

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collector

Konfiqurasiya faylını dəyişdiririk /etc/nginx-log-collector/config.yaml:

  .......
  upload:
    table: nginx.access_log
    dsn: http://klaster-clickhouse-ip:8123/

- tag: "nginx_error:"
  format: error  # access | error
  buffer_size: 1048576
  upload:
    table: nginx.error_log
    dsn: http://klaster-clickhouse-ip:8123/

nginx-in konfiqurasiyası

nginx-in ümumi konfiqurasiyası:

user  nginx;
worker_processes  auto;

#error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format avito_json escape=json
                     '{'
                     '"event_datetime": "$time_iso8601", '
                     '"server_name": "$server_name", '
                     '"remote_addr": "$remote_addr", '
                     '"remote_user": "$remote_user", '
                     '"http_x_real_ip": "$http_x_real_ip", '
                     '"status": "$status", '
                     '"scheme": "$scheme", '
                     '"request_method": "$request_method", '
                     '"request_uri": "$request_uri", '
                     '"server_protocol": "$server_protocol", '
                     '"body_bytes_sent": $body_bytes_sent, '
                     '"http_referer": "$http_referer", '
                     '"http_user_agent": "$http_user_agent", '
                     '"request_bytes": "$request_length", '
                     '"request_time": "$request_time", '
                     '"upstream_addr": "$upstream_addr", '
                     '"upstream_response_time": "$upstream_response_time", '
                     '"hostname": "$hostname", '
                     '"host": "$host"'
                     '}';

    access_log     syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
    error_log      syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse

    #access_log  /var/log/nginx/access.log  main;

    proxy_ignore_client_abort on;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}

Virtual host one:

vhost1.conf:

upstream backend {
    server server-ip-address-with-stub_http_server:8080;
    server server-ip-address-with-stub_http_server:8080;
    server server-ip-address-with-stub_http_server:8080;
    server server-ip-address-with-stub_http_server:8080;
    server server-ip-address-with-stub_http_server:8080;
}

server {
    listen   80;
    server_name vhost1;
    location / {
        proxy_pass http://backend;
    }
}

Add virtual hosts to the file /etc/hosts:

server-ip-address-with-nginx vhost1

HTTP server emulator

We will use as an HTTP server emulator nodejs-stub-server aylıq Maxim Ignatenko

Nodejs-stub-server does not have rpm. Here https://github.com/patsevanton/nodejs-stub-server ona rpm yaradırıq. rpm, Fedora Copr ilə yığılacaq

We install the upstream nginx rpm package nodejs-stub-server

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_server

Load testing

Testing is conducted using Apache benchmark.

We install it:

yum install -y httpd-tools

We start testing using Apache benchmark from 5 different servers:

while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done

Configuring Grafana

You won't find a dashboard on the official Grafana site.

Therefore, we will create it manually.

You can find my saved dashboard burada.

You also need to create a variable table with the contents nginx.access_log.

Singlestat Total Requests:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Failed Requests:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t

Singlestat Failing Percent:

SELECT
 1 as t, (sum(status = 500 or status = 499) / sum(status = 200 or status = 201 or status = 401)) * 100 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Avg Response Time:

SELECT
 1, avg(request_time) FROM $table
 WHERE $timeFilter GROUP BY 1

Singlestat Max Response Time:

SELECT
 1 as t, max(request_time) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Count Status:

$columns(status, count(*) as c) from $table

To display data as a pie chart, you need to install the plugin and restart Grafana.

grafana-cli plugins install grafana-piechart-panel
service grafana-server restart

Pie TOP 5 Status:

SELECT
    1, /* fake timestamp value */
    status,
    sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5

Next, I will provide queries without screenshots:

Count http_user_agent:

$columns(http_user_agent, count(*) c) FROM $table

GoodRate/BadRate:

$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $table

Response Timing:

$rate(avg(request_time) as request_time) FROM $table

Upstream response time (response time of the 1st upstream):

$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $table

Table Count Status for all vhost:

$columns(status, count(*) as c) from $table

Overall view of the dashboard

Comparison of avg() and quantile()

avg()

quantile()

Nəticə:

I hope the community will join in the development/testing and use of nginx-log-collector.
And someone will implement nginx-log-collector and tell how much disk, RAM, CPU has been saved.

Telegram channels:

• Clickhouse
• Nginx
• Church of metrics
• Collection and analytics of system messages
• Grafana

Milliseconds:

If milliseconds matter to you, please write or vote in this issue.

Mənbə: habr.com