🥇 Nginx-log-collector utilitate ab Avito mittendis nginx ligna ad Clickhouse

Hic articulus erit de project nginx-log-collector, quae nginx tigna leget et mittet in botrum Clickhouse. Plerumque ElasticSearch pro lignis adhibetur. Clickhouse minus requirit facultates (spatii orbis, RAM, CPU). Clickhouse records notitia velocius. Clickhouse comprimit notitias, notitias in disco magis compactas faciens. Commoda Clickhouse apparent in 2 prolabitur ex fama Quomodo VK notitias in strepita inserit ex decem milibus servientium.

Ad considerandam analyticam ex lignis, ashboardday pro Grafana creabimus.

Quisque commodo, felis id suscipit.

Instrue nginx, grafana in regula.

Installing a clickhouse botrum portassent utens ansible-playbook e Dionysius Proskurin.

Creando databases ac mensas in Clickhouse

In hoc file SQL queries de datorum creandorum ac tabularum nginx-log-collectoris in Clickhouse describuntur.

Petitionem singulas singulas facimus in botro Clickhouse in servo.

Maximus, note. In hac linea, logs_cluster reponi debet cum botro nomine tuo e fasciculi pressione_remoto_servorum.xml inter "servatores" et "shard".

ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())

Installing and Vestibulum nginx-log-collector-rpm

Nginx-log-collector RPM non habet. Hic https://github.com/patsevanton/nginx-log-collector-rpm RPM partum pro eo. RPM componantur utens Fedora Copr

Install rpm sarcina nginx-log-collector-rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collector

Edere config /etc/nginx-log-collector/config.yaml:

  .......
  upload:
    table: nginx.access_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

- tag: "nginx_error:"
  format: error  # access | error
  buffer_size: 1048576
  upload:
    table: nginx.error_log
    dsn: http://ip-адрес-кластера-clickhouse:8123/

Erexerit nginx

Generalis nginx config;

user  nginx;
worker_processes  auto;

#error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format avito_json escape=json
                     '{'
                     '"event_datetime": "$time_iso8601", '
                     '"server_name": "$server_name", '
                     '"remote_addr": "$remote_addr", '
                     '"remote_user": "$remote_user", '
                     '"http_x_real_ip": "$http_x_real_ip", '
                     '"status": "$status", '
                     '"scheme": "$scheme", '
                     '"request_method": "$request_method", '
                     '"request_uri": "$request_uri", '
                     '"server_protocol": "$server_protocol", '
                     '"body_bytes_sent": $body_bytes_sent, '
                     '"http_referer": "$http_referer", '
                     '"http_user_agent": "$http_user_agent", '
                     '"request_bytes": "$request_length", '
                     '"request_time": "$request_time", '
                     '"upstream_addr": "$upstream_addr", '
                     '"upstream_response_time": "$upstream_response_time", '
                     '"hostname": "$hostname", '
                     '"host": "$host"'
                     '}';

    access_log     syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
    error_log      syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse

    #access_log  /var/log/nginx/access.log  main;

    proxy_ignore_client_abort on;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}

Virtus una virtualis;

vhost1.conf:

upstream backend {
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
    server ip-адрес-сервера-с-stub_http_server:8080;
}

server {
    listen   80;
    server_name vhost1;
    location / {
        proxy_pass http://backend;
    }
}

Virtutes virtuales addere fasciculum /etc/exercituum:

ip-адрес-сервера-с-nginx vhost1

HTTP server aemulator

Ut HTTP server aemula nos utimur nodejs-stub-server ex Maximus Ignatenko

Nodejs-stub-servo rpm non habet. Hic https://github.com/patsevanton/nodejs-stub-server RPM partum pro eo. RPM componantur utens Fedora Copr

Install nodejs-stub servo sarcina in flumine nginx rpm

yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_server

Suspendisse Testis

Portamus experimur Apache Probatio utens.

Instrue eam:

yum install -y httpd-tools

Committitur tentare Apache Probatio ab 5 diversis ministris uti:

while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done

Grafana profecta est

ashboardday in officiali Grafana website non invenies.

Per manus ergo faciemus.

Potes invenire mihi salvus ashboardday hic.

Etiam mensam variabilem cum contentis creare debes nginx.access_log.

Summa supplicum libellis Singlestat:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Deficio supplicum libellis:

SELECT
 1 as t,
 count(*) as c
 FROM $table
 WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t

Deficiente sentio Singlestat:

SELECT
 1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
 WHERE $timeFilter GROUP BY t

Singlestat Avg Response Time:

SELECT
 1, avg(request_time) FROM $table
 WHERE $timeFilter GROUP BY 1

Singlestat Max Response Time:

SELECT
 1 as t, max(request_time) as c
 FROM $table
 WHERE $timeFilter GROUP BY t

Numerare Status:

$columns(status, count(*) as c) from $table

Ad notitias quasi scriblitas outputare, debes plugin instituere et grafana sileo.

grafana-cli plugins install grafana-piechart-panel
service grafana-server restart

Pie TOP 5 Status:

SELECT
    1, /* fake timestamp value */
    status,
    sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5

Praeterea petitiones dabo sine eenshotsscray;

Numera http_user_agent:

$columns(http_user_agent, count(*) c) FROM $table

GoodRate/BadRate:

$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $table

Responsio Timing:

$rate(avg(request_time) as request_time) FROM $table

Responsum adverso tempore (responsio adverso tempore);

$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $table

Mensa comitis Status pro omnibus vhost:

$columns(status, count(*) as c) from $table

General sententia ashboardday

Comparatio avg() et quantitatis ()

AVG()

quantile ()

conclusioni,

Spero communitatem implicari in explicando/temptatione et nginx-log-collectore utendo.
Et cum aliquis instrumentorum nginx-log-collectoris, quantum servaverint in orbe, RAM et CPU, indicabunt tibi.

Canales telegraphi:

Milliseconds:

Ad quos milliseconds re, quaeso, scribe vel suffragium in hoc exitus.

Source: www.habr.com

Nginx-log-collectoris utilitas ex Avito mittendi nginx ligna ad Clickhouse