Dit artikel sil it projekt besprekke , dy't nginx-logs lêze sil, stjoere se nei it Clickhouse-kluster. Meastal wurdt ElasticSearch brûkt foar logs. Clickhouse fereasket minder boarnen (skiifromte, RAM, CPU). Clickhouse skriuwt gegevens rapper. Clickhouse komprimearret de gegevens, wat de gegevens op skiif noch kompakter makket. Foardielen fan Clickhouse kinne sjoen wurde yn 2 dia's út it rapport
Om analytics te besjen troch logs, litte wy in dashboard meitsje foar Grafana.
Wa makket it út, wolkom ûnder kat.
Ynstallearje nginx, grafana op 'e standert manier.
Ynstallearje clickhouse kluster mei ansible-playbook fan .
It meitsjen fan in databank en tabellen yn Clickhouse
Yn dit SQL-fragen foar it meitsjen fan databases en tabellen foar nginx-log-collector yn Clickhouse wurde beskreaun.
Wy meitsje elk fersyk op syn beurt op elke server fan it Clickhouse-kluster.
Wichtige notysje. Yn dizze rigel moat logs_cluster wurde ferfongen troch jo klusternamme út it clickhouse_remote_servers.xml-bestân tusken "remote_servers" en "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())Ynstallearje en konfigurearje nginx-log-collector-rpm
Nginx-log-collector hat gjin rpm. Hjir meitsje rpm foar it. rpm sil wurde boud mei help fan
Ynstallearje rpm-pakket nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorConfig /etc/nginx-log-collector/config.yaml bewurkje:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/It ynstellen fan nginx
Algemiene nginx konfiguraasje:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Firtuele host ien:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}Foegje firtuele hosts ta oan it /etc/hosts-bestân:
ip-адрес-сервера-с-nginx vhost1HTTP tsjinner emulator
As HTTP-serveremulator sille wy brûke от
nodejs-stub-tsjinner hat gjin rpm. Hjir meitsje rpm foar it. rpm sil wurde boud mei help fan
Ynstallearje nodejs-stub-server-pakket op streamop nginx rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverStress Testing
Testen wurdt útfierd mei Apache-benchmark.
Ynstallearje it:
yum install -y httpd-toolsWy begjinne te testen mei Apache-benchmark fan 5 ferskillende servers:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneIt opsetten fan Grafana
Jo sille gjin dashboard fine op 'e offisjele Grafana-webside.
Dêrom sille wy it mei de hân dwaan.
Jo kinne myn bewarre dashboard fine .
Jo moatte ek in tabelfariabele meitsje mei de ynhâld nginx.access_log.
Singlestat Totaal oanfragen:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Singlestat mislearre fersiken:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Singlestat Failing Persintaazje:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Singlestat Gemiddelde reaksjetiid:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Singlestat Max Response Tiid:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Telstatus:
$columns(status, count(*) as c) from $table
Om gegevens út te jaan as in taart, moatte jo de plugin ynstallearje en grafana opnij laden.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartTaart TOP 5 Status:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
Fierder sil ik oanfragen sûnder skermôfbyldings jaan:
Tel http_user_agent:
$columns(http_user_agent, count(*) c) FROM $tableGoodRate/BadRate:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tablereaksje timing:
$rate(avg(request_time) as request_time) FROM $tableStroomopwaartse reaksjetiid (antwurdtiid fan de 1e streamopop):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableStatus tabeltelling foar alle vhosts:
$columns(status, count(*) as c) from $tableAlgemiene werjefte fan it dashboard
Fergelykje avg() en quantile()
avg()
kwantyl ()
Fermelding:
Hooplik sil de mienskip belutsen wurde by it ûntwikkeljen / testen en brûken fan nginx-log-collector.
En as immen nginx-log-collector ymplemintearret, sil hy jo fertelle hoefolle hy hat bewarre skiif, RAM, CPU.
Telegramkanalen:
Millisekonden:
Wa soarget foar millisekonden, skriuw of stim, asjebleaft, yn dizze .
Boarne: www.habr.com