🥇Nginx-log-collector utility kubva kuAvito yekutumira nginx matanda kuClickhouse

Nyaya ino ichakurukura nezvebasa racho nginx-log-collector, iyo inoverenga nginx matanda uye itumire kune iyo Clickhouse cluster. Kazhinji ElasticSearch inoshandiswa pamatanda. Clickhouse inoda zvishoma zviwanikwa (dhisiki nzvimbo, RAM, CPU). Clickhouse inorekodha data nekukurumidza. Clickhouse inomanikidza dhata, ichiita iyo data pa diski iwedzere compact. Zvakanakira zveClickhouse zvinoonekwa mumasiraidhi maviri kubva kumushumo VK inoisa sei data muClickHouse kubva kumakumi ezviuru emaseva.

Kuti uone analytics yakavakirwa pamatanda, isu tichagadzira dashboard yeGrafana.

Chero ani anofarira, anogamuchirwa kukati.

Isa nginx, grafana nenzira yakajairika.

Kuisa a clickhouse cluster uchishandisa ansible-playbook kubva Denis Proskurin.

Kugadzira dhatabhesi uye matafura muClickhouse

Mune izvozvo faira SQL mibvunzo yekugadzira dhatabhesi uye matafura e nginx-log-muunganidzi muClickhouse anotsanangurwa.

Isu tinoita chikumbiro chimwe nechimwe pane yega sevha muClickhouse cluster.

Chinyorwa chakakosha. Mumutsara uyu, logs_cluster inoda kutsiviwa nezita rechikwata chako kubva paclickhouse_remote_servers.xml faira pakati pe "remote_servers" ne "shard".

ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())

Kuisa uye kugadzirisa nginx-log-collector-rpm

Nginx-log-collector haina rpm. Here https://github.com/patsevanton/nginx-log-collector-rpm gadzira rpm kwayo. rpm ichaunganidzwa uchishandisa Fedora Copr

Isa iyo rpm package nginx-log-collector-rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collector

Rongedza iyo config /etc/nginx-log-collector/config.yaml:

  .......
  upload:
    table: nginx.access_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

- tag: "nginx_error:"
  format: error  # access | error
  buffer_size: 1048576
  upload:
    table: nginx.error_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

Kugadzika nginx

General nginx config:

user  nginx;
worker_processes  auto;

#error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format avito_json escape=json
                     '{'
                     '"event_datetime": "$time_iso8601", '
                     '"server_name": "$server_name", '
                     '"remote_addr": "$remote_addr", '
                     '"remote_user": "$remote_user", '
                     '"http_x_real_ip": "$http_x_real_ip", '
                     '"status": "$status", '
                     '"scheme": "$scheme", '
                     '"request_method": "$request_method", '
                     '"request_uri": "$request_uri", '
                     '"server_protocol": "$server_protocol", '
                     '"body_bytes_sent": $body_bytes_sent, '
                     '"http_referer": "$http_referer", '
                     '"http_user_agent": "$http_user_agent", '
                     '"request_bytes": "$request_length", '
                     '"request_time": "$request_time", '
                     '"upstream_addr": "$upstream_addr", '
                     '"upstream_response_time": "$upstream_response_time", '
                     '"hostname": "$hostname", '
                     '"host": "$host"'
                     '}';

    access_log     syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
    error_log      syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse

    #access_log  /var/log/nginx/access.log  main;

    proxy_ignore_client_abort on;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}

Imwe virtual host:

vhost1.conf:

upstream backend {
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
}

server {
    listen   80;
    server_name vhost1;
    location / {
        proxy_pass http://backend;
    }
}

Wedzera virtual host kune iyo /etc/hosts faira:

ip-адрес-сервера-с-nginx vhost1

HTTP server emulator

Seye HTTP sevha emulator isu tichashandisa nodejs-stub-server от Maxim Ignatenko

Nodejs-stub-server haina rpm. Here https://github.com/patsevanton/nodejs-stub-server gadzira rpm kwayo. rpm ichaunganidzwa uchishandisa Fedora Copr

Isa nodejs-stub-server package pane kumusoro nginx rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_server

Stress Testing

Isu tinoita bvunzo tichishandisa Apache bhenji.

Isa iyo:

yum install -y httpd-tools

Isu tinotanga kuyedza tichishandisa Apache bhenji kubva kune mashanu akasiyana maseva:

while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done

Kugadzira Grafana

Iwe hauzowani dashboard pane yepamutemo Grafana webhusaiti.

Naizvozvo, tichazviita nemaoko.

Unogona kuwana dashboard yangu yakachengetwa pano.

Iwe unofanirwawo kugadzira tafura yakasiyana nezviri mukati nginx.access_log.

Singlestat Total Zvikumbiro:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Zvakundikana Zvikumbiro:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t

Singlestat Failing Percent:

SELECT
 1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Avg Nguva Yekupindura:

SELECT
 1, avg(request_time) FROM $table
 WHERE $timeFilter GROUP BY 1

Singlestat Max Mhinduro Nguva:

SELECT
 1 as t, max(request_time) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Count Status:

$columns(status, count(*) as c) from $table

Kuti ubudise data senge pie, unofanirwa kuisa iyo plugin uye wotangazve grafana.

grafana-cli plugins install grafana-piechart-panel
service grafana-server restart

Pie TOP 5 Mamiriro:

SELECT
    1, /* fake timestamp value */
    status,
    sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5

Uyezve ini ndichapa zvikumbiro pasina skrini:

Verenga http_mushandisi_mumiririri:

$columns(http_user_agent, count(*) c) FROM $table

GoodRate/BadRate:

$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $table

Nguva Yemhinduro:

$rate(avg(request_time) as request_time) FROM $table

Nguva yekupindura yakakwira (yekutanga yekupindura nguva):

$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $table

Tafura Yekuverenga Mamiriro kune ese vhost:

$columns(status, count(*) as c) from $table

General maonero edashboard

Kuenzanisa kwe avg () uye quantile ()

avg()

quantile()

Mhedziso:

Ndinovimba nharaunda ichabatanidzwa mukugadzira / kuyedza uye kushandisa nginx-log-collector.
Uye kana mumwe munhu ashandisa nginx-log-collector, ivo vanokuudza kuti yakawanda sei yavakachengeta pa diski, RAM, uye CPU.

Zviteshi zveTeregiramu:

Masekondi

Kune akakosha mamilliseconds, ndapota nyora kana kuvhota mune izvi nyaya.

Source: www.habr.com

Nginx-log-collector utility kubva kuAvito yekutumira nginx matanda kuClickhouse