Sehlooho sena se tla tšohla morero
Ho sheba analytics ka logs, ha re theheng dashboard bakeng sa Grafana.
Ke mang ea tsotellang, amohela tlas'a katse.
Kenya nginx, grafana ka tsela e tloaelehileng.
Kenya sehlopha sa Clickhouse ka ansible-playbook ho tloha
Ho theha database le litafole ho Clickhouse
Ho seo
Re etsa kopo ka 'ngoe ho seva se seng le se seng sa sehlopha sa Clickhouse.
Molaetsa oa bohlokoa. Moleng ona, logs_cluster e tlameha ho nkeloa sebaka ka lebitso la sehlopha sa hau ho tsoa ho faele ea clickhouse_remote_servers.xml lipakeng tsa "remote_servers" le "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())
Ho kenya le ho lokisa nginx-log-collector-rpm
Nginx-log-collector ha e na rpm. Mona
Kenya sephutheloana sa rpm nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collector
Fetola tlhophiso /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
Ho theha nginx
Kakaretso ea nginx config:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Virtual host one:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}
Eketsa li-host host ho file ea /etc/hosts:
ip-адрес-сервера-с-nginx vhost1
Emulator ea seva ea HTTP
Joaloka emulator ea seva sa HTTP re tla e sebelisa
nodejs-stub-server ha e na rpm. Mona
Kenya sephutheloana sa nodejs-stub-server sebakeng se ka holimo sa nginx rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_server
Teko ea Khatello ea Maikutlo
Teko e etsoa ho sebelisoa benchmark ea Apache.
E kenye:
yum install -y httpd-tools
Re qala ho leka ho sebelisa benchmark ea Apache ho tsoa ho li-server tse 5 tse fapaneng:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
Ho theha Grafana
U ke ke ua fumana dashboard webosaeteng ea semmuso ea Grafana.
Ka hona, re tla e etsa ka letsoho.
U ka fumana dashboard ea ka e bolokiloeng
U boetse u hloka ho theha tafole e fapaneng e nang le litaba nginx.access_log
.
Singlestat Kakaretso ea Likopo:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Likopo tse Hlolehileng tsa Singlestat:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Singlestat Failing Percent:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Nako ea Karabo ea Singlestat Avg:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Nako ea Karabo ea Singlestat Max:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Boemo ba ho Bala:
$columns(status, count(*) as c) from $table
Ho hlahisa data joalo ka phae, o hloka ho kenya plugin le ho kenya grafana hape.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restart
Boemo ba Pie TOP 5:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
Ho feta moo ke tla fana ka likopo ntle le li-screenshots:
Bala http_user_agent:
$columns(http_user_agent, count(*) c) FROM $table
GoodRate/BadRate:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $table
nako ea karabo:
$rate(avg(request_time) as request_time) FROM $table
Nako ea karabelo holimo (nako ea karabelo ea 1st upstream):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $table
Tafole Count Status bakeng sa li-vhosts tsohle:
$columns(status, count(*) as c) from $table
Pono e akaretsang ea dashboard
Ho bapisa avg() le quantile()
kakaretso()
quantile()
Qetello:
Re tšepa hore sechaba se tla kenya letsoho ho nts'etsopele/tekong le ho sebelisa nginx-log-collector.
'Me ha motho a sebelisa nginx-log-collector, o tla u bolella hore na o bolokile disk, RAM, CPU bokae.
Liteishene tsa Telegraph:
Milliseconds:
Ke mang ea tsotellang milliseconds, ngola kapa o vouta, ka kopo, ho sena
Source: www.habr.com