ืืขืจ ืึทืจืืืงื ืืืขื ืืืกืงืืืืจื ืื ืคึผืจืืืขืงื , ืืืึธืก ืืืขื ืืืืขื ืขื nginx ืืึธืืก ืืื ืฉืืงื ืืื ืฆื ืื ืงืืืงืืึธืืกืข ืงื ืืื. ืืืืฉืึทืืืึทืื ElasticSearch ืืื ืืขื ืืฆื ืคึฟืึทืจ ืืึธืืก. ืงืืืงืืึธืืกืข ืจืืงืืืืืขืจื ืืืืื ืืงืขืจ ืจืขืกืืจืกื (ืืืกืง ืคึผืืึทืฅ, ืืึทืจืึทื, ืงืคึผื). ืงืืืงืืึธืืกืข ืจืขืงืึธืจืืก ืืึทืื ืคืึทืกืืขืจ. ืงืืืงืืึธืืกืข ืงืึทืืคึผืจืขืกืื ืืึทืื, ืืืื ืื ืืึทืื ืืืืฃ ืืืกืง ืืคืืื ืืขืจ ืกืึธืืื. ืื ืึทืืืืึทื ืืืืืฉืื ืคืื Clickhouse ืืขื ืขื ืงืขื ืืืง ืืื 2 ืกืืืืื ืคืื ืืขื ืืึทืจืืื
ืฆื ืืขื ืึทื ืึทืืืืืงืก ืืืืืจื ืืืืฃ ืืึธืืก, ืืืจ ืืึทืื ืึท ืืึทืฉืืึธืจื ืคึฟืึทืจ Grafana.
ืืืขืจ ืขืก ืื ืืื ืืขืจืขืกืืจื, ืืึทืืจืืกืื ื ืฆื ืงืึทืฅ.
ืื ืกืืึทืืืจื nginx, grafana ืืื ืื ื ืึธืจืืึทื ืืืขื.
ืื ืกืืึธืืื ื ืึท ืงืืืงืืึธืืกืข ืงื ืืื ื ืืฆื ืึทื ืกืืืืข-ืคึผืืืึทืืึธืึธืง ืคึฟืื .
ืงืจืืืืืื ื ืืึทืืึทืืืืกืื ืืื ืืืฉื ืืื ืงืืืงืงืืึธืืกืข
ืืื ืืขื ืกืงื ืงืืืืจืื ืคึฟืึทืจ ืงืจืืืืืื ื ืืึทืืึทืืืืกืื ืืื ืืืฉื ืคึฟืึทืจ nginx-log-collector ืืื Clickhouse ืืขื ืขื ืืืกืงืจืืืื.
ืืืจ ืืึทืื ืืขืืขืจ ืืงืฉื ืืืื ืขืจ ืืืจื ืืืื ืขืจ ืืืืฃ ืืขืืขืจ ืกืขืจืืืขืจ ืืื ืื ืงืืืงืืึธืืกืข ืงื ืืื.
ืืืืืืืง ืืึธื. ืืื ืืขื ืฉืืจื, ืืึธืืก_ืงืืืกืืขืจ ืืึทืจืฃ ืืืื ืจืืคึผืืืืกื ืืื ืืืื ืงื ืืื ื ืึธืืขื ืคืื ืื ืงืืืงืืึธืืกืข_ืจืขืืึธืืข_ืกืขืจืืืขืจืก.ืงืกืื ืืขืงืข ืฆืืืืฉื "ืจืืืึธืื_ืกืขืจืืืขืจืก" ืืื "ืฉืึทืจื".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())ืื ืกืืึธืืื ื ืืื ืงืึทื ืคืืืืขืจ nginx-log-collector-rpm
Nginx-log-collector ืืื ื ืืฉื ืึท ืจืคึผื. ืืึธ ืฉืึทืคึฟื ืจืคึผื ืคึฟืึทืจ ืขืก. rpm ืืืขื ืืืื ืงืึทืืคึผืืืื ืืื
ืื ืกืืึทืืืจื ืื rpm ืคึผืขืงื nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorืจืขืืึทืืืจื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-ะฐะดัะตั-ะบะปะฐััะตัะฐ-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-ะฐะดัะตั-ะบะปะฐััะตัะฐ-clickhouse:8123/ืืึทืฉืืขืืืงื nginx
ืึทืืืขืืืื ืข nginx ืงืึทื ืคืืืืขืจืืืฉืึทื:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
ืืืื ืืืืจืืืขื ืืึทืืขืืึธืก:
vhost1.conf:
upstream backend {
server ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-stub_http_server:8080;
server ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-stub_http_server:8080;
server ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-stub_http_server:8080;
server ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-stub_http_server:8080;
server ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}ืืืื ืืืืจืืืึทื ืืื ืืช ืฆื ืื /etc/hosts ืืขืงืข:
ip-ะฐะดัะตั-ัะตัะฒะตัะฐ-ั-nginx vhost1ืืืืคึผ ืกืขืจืืืขืจ ืขืืืืึทืืึธืจ
ืืื ืึท ืืืืคึผ ืกืขืจืืืขืจ ืขืืืืึทืืึธืจ ืืืจ ืืืขืื ื ืืฆื ืคืื
Nodejs-stub-server ืืื ื ืืฉื ืึท ืจืคึผื. ืืึธ ืฉืึทืคึฟื ืจืคึผื ืคึฟืึทืจ ืขืก. rpm ืืืขื ืืืื ืงืึทืืคึผืืืื ืืื
ืื ืกืืึทืืืจื nodejs-stub-server ืคึผืขืงื ืืืืฃ ืึทืคึผืกืืจืื nginx ืจืคึผื
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverืืจืืง ืืขืกืืื ื
ืืืจ ืืืจืืคืืจื ืืขืกืืื ื ืืื ืึทืคึผืึทืืฉื ืืขื ืืฉืืึทืจืง.
ืื ืกืืึทืืืจื ืขืก:
yum install -y httpd-toolsืืืจ ืึธื ืืืืื ืืขืกืืื ื ืืื ืึทืคึผืึทืืฉื ืืขื ืืฉืืึทืจืง ืคึฟืื 5 ืคืึทืจืฉืืืขื ืข ืกืขืจืืืขืจืก:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneืืึทืฉืืขืืืงื ืืจืึทืคืึทื ืึท
ืืืจ ืืืขื ื ืื ืืขืคึฟืื ืขื ืึท ืืึทืฉืืึธืจื ืืืืฃ ืืขืจ ืืึทืึทืืืขืจ Grafana ืืืขืืืืืื.
ืืขืจืืืขืจ, ืืืจ ืืืขืื ืืึธื ืขืก ืืื ืืึทื ื.
ืืืจ ืงืขื ืขื ืืขืคึฟืื ืขื ืืืื ืืขืจืืืขืืืขื ืืึทืฉืืึธืจื .
ืืืจ ืืืื ืืึทืจืคึฟื ืฆื ืฉืึทืคึฟื ืึท ืืืฉ ืืืึทืืขืืืืืง ืืื ืื ืืื ืืึทืื nginx.access_log.
ืกืื ืืืขืกืืึทื ืืึทื ืฅ ืจืืงืืืขืก:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
ืกืื ืืืขืกืืึทื ืคืืืื ืจืืงืืืขืก:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
ืกืื ืืืขืกืืึทื ืคืืืืื ื ืคึผืจืึธืฆืขื ื:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
ืกืื ืืืขืกืืึทื ืืืจืืฉื ืืืืขื ืขื ืืคืขืจ ืฆืืื:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
ืกืื ืืืขืกืืึทื ืืึทืงืก ืขื ืืคืขืจ ืฆืืื:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
ืฆืืืื ืกืืึทืืืก:
$columns(status, count(*) as c) from $table
ืฆื ืจืขืืืืืึทื ืืึทืื ืืื ืึท ืคึผืืจืึธื, ืืืจ ืืึทืจืคึฟื ืฆื ืื ืกืืึทืืืจื ืื ืคึผืืืืื ืืื ืจืืกืืึทืจื ืืจืึทืคืึทื ืึท.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartืคึผืืจืึธื ืฉืคึผืืฅ 5 ืกืืึทืืืก:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
ืืืืึทืืขืจ ืืื ืืืขื ืืขืื ืจืืงืืืขืก ืึธื ืกืงืจืขืขื ืฉืึธืฅ:
ืฆืืืื http_user_agent:
$columns(http_user_agent, count(*) c) FROM $tableGoodRate/BadRate:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tableืขื ืืคืขืจ ืฆืืื:
$rate(avg(request_time) as request_time) FROM $tableืึทืคึผืกืืจืื ืขื ืืคืขืจ ืฆืืื (1st ืึทืคึผืกืืจืื ืขื ืืคืขืจ ืฆืืื):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableืืึทืืืข ืืจืืฃ ืกืืึทืืืก ืคึฟืึทืจ ืึทืืข ืืืืึธืกื:
$columns(status, count(*) as c) from $tableืึทืืืขืืืื ืข ืืืื ืื ื ืคืื ืื ืืึทืฉืืึธืจื
ืคืึทืจืืืืึทื ืคืื ืึทืืื () ืืื ืงืืืึทื ืืื ()
avg()
ืงืืืึทื ืืื ()
ืืกืงื ื:
ืืื ืืึธืคึฟื ืื ืงืื ืืื ืื ืืืึทืืืื ืืื ืืขืืืขืืึธืคึผืื ื / ืืขืกืืื ื ืืื ื ืืฆื nginx-log-collector.
ืืื ืืืขื ืขืืขืฆืขืจ ืืืคึผืืึทืืึทื ืฅ nginx-log-collector, ืืื ืืืขืื ืืึธืื ืืืจ ืืื ืคืื ืืืกืง, ืืึทืจืึทื ืืื ืงืคึผื ืืื ืืขืจืืืขืืืขื.
ืืขืืขืืจืึทื ืืฉืึทื ืึทืื:
ืืืืืกืขืงืื ืื:
ืฆื ืืืขืืขื ืืืืืกืขืงืึทื ืื ืขื ืื, ืืืืข ืฉืจืืึทืื ืึธืืขืจ ืฉืืืืขื ืืื ืืขื .
ืืงืืจ: www.habr.com