🥇Huduma ya ukusanyaji wa logi ya Nginx kutoka Avito kwa kutuma kumbukumbu za nginx kwa Clickhouse

Nakala hii itajadili mradi huo mtoza-logi wa nginx, ambayo itasoma kumbukumbu za nginx, itume kwa nguzo ya Clickhouse. Kawaida ElasticSearch hutumiwa kwa kumbukumbu. Clickhouse inahitaji rasilimali chache (nafasi ya diski, RAM, CPU). Clickhouse huandika data haraka zaidi. Clickhouse inabana data, ambayo hufanya data kwenye diski kuwa ngumu zaidi. Faida za Clickhouse zinaweza kuonekana katika slaidi 2 kutoka kwa ripoti Jinsi VK inaingiza data kwenye ClickHouse kutoka kwa makumi ya maelfu ya seva.

Ili kuona uchanganuzi kulingana na kumbukumbu, hebu tuunde dashibodi ya Grafana.

Nani anajali, karibu chini ya paka.

Sakinisha nginx, grafana kwa njia ya kawaida.

Sakinisha nguzo ya kubofya kwa kutumia kitabu cha kucheza kutoka Denis Proskurin.

Kuunda hifadhidata na meza katika Clickhouse

Katika hii faili Maswali ya SQL ya kuunda hifadhidata na jedwali za mtoza-logi wa nginx katika Clickhouse yameelezewa.

Tunafanya kila ombi kwa zamu kwa kila seva ya nguzo ya Clickhouse.

Ujumbe muhimu. Katika mstari huu, logs_cluster inapaswa kubadilishwa na jina la nguzo yako kutoka faili ya clickhouse_remote_servers.xml kati ya "remote_servers" na "shard".

ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())

Kufunga na kusanidi nginx-log-collector-rpm

Nginx-log-collector haina rpm. Hapa https://github.com/patsevanton/nginx-log-collector-rpm tengeneza rpm kwa ajili yake. rpm itajengwa kwa kutumia Fedora Copr

Sakinisha kifurushi cha rpm nginx-log-collector-rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collector

Hariri usanidi /etc/nginx-log-collector/config.yaml:

  .......
  upload:
    table: nginx.access_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

- tag: "nginx_error:"
  format: error  # access | error
  buffer_size: 1048576
  upload:
    table: nginx.error_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

Kuanzisha nginx

Usanidi wa jumla wa nginx:

user  nginx;
worker_processes  auto;

#error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format avito_json escape=json
                     '{'
                     '"event_datetime": "$time_iso8601", '
                     '"server_name": "$server_name", '
                     '"remote_addr": "$remote_addr", '
                     '"remote_user": "$remote_user", '
                     '"http_x_real_ip": "$http_x_real_ip", '
                     '"status": "$status", '
                     '"scheme": "$scheme", '
                     '"request_method": "$request_method", '
                     '"request_uri": "$request_uri", '
                     '"server_protocol": "$server_protocol", '
                     '"body_bytes_sent": $body_bytes_sent, '
                     '"http_referer": "$http_referer", '
                     '"http_user_agent": "$http_user_agent", '
                     '"request_bytes": "$request_length", '
                     '"request_time": "$request_time", '
                     '"upstream_addr": "$upstream_addr", '
                     '"upstream_response_time": "$upstream_response_time", '
                     '"hostname": "$hostname", '
                     '"host": "$host"'
                     '}';

    access_log     syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
    error_log      syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse

    #access_log  /var/log/nginx/access.log  main;

    proxy_ignore_client_abort on;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}

Mpangishi halisi wa kwanza:

vhost1.conf:

upstream backend {
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
}

server {
    listen   80;
    server_name vhost1;
    location / {
        proxy_pass http://backend;
    }
}

Ongeza majeshi ya kawaida kwa /etc/hosts faili:

ip-адрес-сервера-с-nginx vhost1

Kiigaji cha seva ya HTTP

Kama emulator ya seva ya HTTP tutatumia nodejs-stub-server kutoka Maxim Ignatenko

nodejs-stub-server haina rpm. Hapa https://github.com/patsevanton/nodejs-stub-server tengeneza rpm kwa ajili yake. rpm itajengwa kwa kutumia Fedora Copr

Sakinisha kifurushi cha nodejs-stub-server kwenye mkondo wa nginx rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_server

Mtihani wa Stress

Upimaji unafanywa kwa kutumia alama ya Apache.

Isakinishe:

yum install -y httpd-tools

Tunaanza kujaribu kwa kutumia alama ya Apache kutoka kwa seva 5 tofauti:

while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done

Kuanzisha Grafana

Hutapata dashibodi kwenye tovuti rasmi ya Grafana.

Kwa hiyo, tutafanya kwa mkono.

Unaweza kupata dashibodi yangu iliyohifadhiwa hapa.

Unahitaji pia kuunda muundo wa jedwali na yaliyomo nginx.access_log.

Jumla ya Maombi ya Singlestat:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Maombi Yanayoshindikana ya Singlestat:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t

Asilimia ya Kushindwa kwa Singlestat:

SELECT
 1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
 WHERE $timeFilter GROUP BY t

Wakati wa Kujibu Wastani wa Singlestat:

SELECT
 1, avg(request_time) FROM $table
 WHERE $timeFilter GROUP BY 1

Muda wa Juu wa Kujibu wa Singlestat:

SELECT
 1 as t, max(request_time) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Hali ya Hesabu:

$columns(status, count(*) as c) from $table

Ili kutoa data kama pai, unahitaji kusakinisha programu-jalizi na upakie upya grafana.

grafana-cli plugins install grafana-piechart-panel
service grafana-server restart

Pie TOP 5 Hali:

SELECT
    1, /* fake timestamp value */
    status,
    sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5

Zaidi nitatoa maombi bila picha za skrini:

Hesabu http_mtumiaji_wakala:

$columns(http_user_agent, count(*) c) FROM $table

Kiwango kizuri/Kiwango kibaya:

$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $table

muda wa majibu:

$rate(avg(request_time) as request_time) FROM $table

Muda wa kujibu wa juu (muda wa kujibu wa mkondo wa 1 wa juu):

$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $table

Hali ya Hesabu ya Jedwali kwa vikali wote:

$columns(status, count(*) as c) from $table

Mtazamo wa jumla wa dashibodi

Kulinganisha wastani () na quantile()

wastani ()

quantile()

Hitimisho:

Tunatumahi kuwa jamii itahusika katika kukuza/kujaribu na kutumia nginx-log-collector.
Na wakati mtu anatumia nginx-log-collector, atakuambia ni kiasi gani alihifadhi disk, RAM, CPU.

Vituo vya Telegraph:

Milisekunde:

Nani anajali kuhusu milliseconds, kuandika au kupiga kura, tafadhali, katika hili suala.

Chanzo: mapenzi.com

Ukusanyaji wa Nginx-logi kutoka kwa Avito kwa kutuma kumbukumbu za nginx kwa Clickhouse