Hierdie artikel sal die projek bespreek , wat nginx-logboeke sal lees, stuur dit na die Clickhouse-kluster. Gewoonlik word ElasticSearch vir logs gebruik. Clickhouse benodig minder hulpbronne (skyfspasie, RAM, SVE). Clickhouse skryf data vinniger. Clickhouse komprimeer die data, wat die data op skyf selfs meer kompak maak. Die voordele van Clickhouse kan in 2 skyfies uit die verslag gesien word
Kom ons skep 'n kontroleskerm vir Grafana om ontledings volgens logs te sien.
Wie gee om, welkom onder kat.
Installeer nginx, grafana op die standaard manier.
Installeer clickhouse cluster met ansible-playbook vanaf .
Skep 'n databasis en tabelle in Clickhouse
Hierin SQL-navrae vir die skep van databasisse en tabelle vir nginx-log-collector in Clickhouse word beskryf.
Ons rig elke versoek om die beurt op elke bediener van die Clickhouse-groepering.
Belangrike nota. In hierdie reël moet logs_cluster vervang word met jou groepnaam van die clickhouse_remote_servers.xml-lêer tussen "remote_servers" en "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())Installeer en konfigureer nginx-log-collector-rpm
Nginx-log-collector het nie 'n rpm nie. Hier skep rpm daarvoor. rpm sal gebou word met behulp van
Installeer rpm-pakket nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorWysig config /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/Stel nginx op
Algemene nginx-opstelling:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Virtuele gasheer een:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}Voeg virtuele gashere by die /etc/hosts-lêer:
ip-адрес-сервера-с-nginx vhost1HTTP-bediener-emulator
As 'n HTTP-bediener-emulator sal ons gebruik van
nodejs-stub-bediener het nie rpm nie. Hier skep rpm daarvoor. rpm sal gebou word met behulp van
Installeer nodejs-stub-bediener pakket op stroomop nginx rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverStrestoetsing
Toetsing word uitgevoer met behulp van Apache-benchmark.
Installeer dit:
yum install -y httpd-toolsOns begin toets met Apache-benchmark vanaf 5 verskillende bedieners:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneGrafana opstel
Jy sal nie 'n dashboard op die amptelike Grafana-webwerf kry nie.
Daarom sal ons dit met die hand doen.
Jy kan my gestoorde dashboard vind .
Jy moet ook 'n tabelveranderlike met die inhoud skep nginx.access_log.
Enkelstaat totale versoeke:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Singlestat mislukte versoeke:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Singlestat druippersentasie:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Singlestat Gemiddelde reaksietyd:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Singlestat maksimum reaksietyd:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Tel Status:
$columns(status, count(*) as c) from $table
Om data soos 'n pastei uit te voer, moet jy die inprop installeer en grafana herlaai.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartTert TOP 5 Status:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
Verder sal ek versoeke sonder skermkiekies gee:
Tel http_user_agent:
$columns(http_user_agent, count(*) c) FROM $tableGoeie koers/Slegte koers:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tablereaksie tydsberekening:
$rate(avg(request_time) as request_time) FROM $tableStroomop reaksietyd (reaksietyd van die 1ste stroomop):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableTabeltellingstatus vir alle vhosts:
$columns(status, count(*) as c) from $tableAlgemene aansig van die dashboard
Vergelyk avg() en quantile()
gem.()
kwantiel()
Gevolgtrekking:
Hopelik sal die gemeenskap betrokke raak by die ontwikkeling/toetsing en gebruik van nginx-log-collector.
En wanneer iemand nginx-log-collector implementeer, sal hy jou vertel hoeveel hy skyf, RAM, CPU gestoor het.
Telegram kanale:
Millisekondes:
Wie gee om vir millisekondes, skryf of stem, asseblief, hierin .
Bron: will.com