Atik sa a pral diskite sou pwojè a , ki pral li mòso nginx epi voye yo bay gwoup Clickhouse la. Anjeneral yo itilize ElasticSearch pou mòso bwa. Clickhouse mande mwens resous (espas disk, RAM, CPU). Clickhouse anrejistre done pi vit. Clickhouse konprese done, fè done yo sou disk menm plis kontra enfòmèl ant. Avantaj Clickhouse yo vizib nan 2 diapositives nan rapò a
Pou wè analiz ki baze sou mòso bwa, nou pral kreye yon tablodbò pou Grafana.
Nenpòt moun ki enterese, akeyi nan chat.
Enstale nginx, grafana nan fason estanda a.
Enstale yon gwoup clickhouse lè l sèvi avèk ansible-playbook soti nan .
Kreye baz done ak tab nan Clickhouse
Nan sa Yo dekri demann SQL pou kreye baz done ak tab pou nginx-log-collector nan Clickhouse.
Nou fè chak demann youn pa youn sou chak sèvè nan gwoup Clickhouse la.
Nòt enpòtan. Nan liy sa a, logs_cluster bezwen ranplase ak non gwoup ou a nan dosye clickhouse_remote_servers.xml ant "remote_servers" ak "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())Enstale ak konfigirasyon nginx-log-collector-rpm
Nginx-log-collector pa gen yon rpm. Isit la kreye rpm pou li. rpm yo pral konpile lè l sèvi avèk
Enstale pake rpm nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorEdite konfigirasyon /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/Mete kanpe nginx
Jeneral nginx konfigirasyon:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Yon lame vityèl:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}Ajoute lame vityèl nan dosye /etc/hosts la:
ip-адрес-сервера-с-nginx vhost1HTTP sèvè Emulation
Kòm yon Emulation sèvè HTTP nou pral itilize soti nan
Nodejs-stub-server pa gen yon rpm. Isit la kreye rpm pou li. rpm yo pral konpile lè l sèvi avèk
Enstale pake nodejs-stub-server sou en nginx rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverTès estrès
Nou fè tès lè l sèvi avèk Apache referans.
Enstale li:
yum install -y httpd-toolsNou kòmanse fè tès lè l sèvi avèk Apache referans nan 5 sèvè diferan:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneMete kanpe Grafana
Ou p ap jwenn yon tablodbò sou sit entènèt ofisyèl Grafana.
Se poutèt sa, nou pral fè li alamen.
Ou ka jwenn tablodbò sove mwen an .
Ou bezwen tou kreye yon varyab tab ak sa ki ladan yo nginx.access_log.
Singlestat Total Demann:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Singlestat Demann ki echwe:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Pousantaj Singlestat echèk:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Tan repons mwayèn Singlestat:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Tan repons Singlestat Max:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Estati konte:
$columns(status, count(*) as c) from $table
Pou pwodiksyon done tankou yon tat, ou bezwen enstale Plugin la epi rekòmanse grafana.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartTat TOP 5 Estati:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
Pli lwen mwen pral bay demann san yo pa Ekran:
Konte http_user_agent:
$columns(http_user_agent, count(*) c) FROM $tableBon To/BadRate:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tableTan repons:
$rate(avg(request_time) as request_time) FROM $tableTan repons en (1ye tan repons en):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableTablo Konte Estati pou tout vhost:
$columns(status, count(*) as c) from $tableView jeneral nan tablodbò a
Konparezon avg () ak quantile ()
mwayèn ()
quantile()
Konklizyon:
Mwen espere ke kominote a pral patisipe nan devlope/tès ak itilize nginx-log-collector.
Epi lè yon moun aplike nginx-log-collector, yo pral di w konbyen yo te sove sou disk, RAM, ak CPU.
Chanèl Telegram:
Milisgond:
Pou ki moun milisgond enpòtan, tanpri ekri oswa vote nan sa a .
Sous: www.habr.com