Nkhaniyi ifotokoza za ntchitoyi , yomwe idzawerenge zipika za nginx ndikuzitumiza ku gulu la Clickhouse. Nthawi zambiri ElasticSearch imagwiritsidwa ntchito pazolemba. Clickhouse imafuna zinthu zochepa (malo a disk, RAM, CPU). Clickhouse amalemba zambiri mwachangu. Clickhouse compresses deta, kupanga deta pa litayamba kukhala yaying'ono. Ubwino wa Clickhouse umawoneka muzithunzi za 2 kuchokera ku lipotilo
Kuti muwone ma analytics potengera matabwa, tipanga dashboard ya Grafana.
Aliyense amene ali ndi chidwi, alandilidwa kwa mphaka.
Ikani nginx, grafana m'njira yokhazikika.
Kukhazikitsa gulu la clickhouse pogwiritsa ntchito ansible-playbook kuchokera .
Kupanga nkhokwe ndi matebulo mu Clickhouse
Mmenemo Mafunso a SQL opangira ma database ndi matebulo a nginx-log-collector ku Clickhouse akufotokozedwa.
Timapempha aliyense payekhapayekha pa seva iliyonse mugulu la Clickhouse.
Mfundo yofunika. Pamzerewu, logs_cluster ikufunika kusinthidwa ndi dzina lagulu lanu kuchokera pa fayilo ya clickhouse_remote_servers.xml pakati pa "remote_servers" ndi "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())Kuyika ndi kukonza nginx-log-collector-rpm
Nginx-log-collector alibe rpm. Pano pangani rpm kwa izo. rpm idzapangidwa pogwiritsa ntchito
Ikani phukusi la rpm nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorSinthani config /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/Kupanga nginx
General nginx config:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Mmodzi wapagulu:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}Onjezani makamu enieni ku fayilo / etc/hosts:
ip-адрес-сервера-с-nginx vhost1HTTP seva emulator
Monga emulator ya seva ya HTTP tidzagwiritsa ntchito от
Nodejs-stub-server ilibe rpm. Pano pangani rpm kwa izo. rpm idzapangidwa pogwiritsa ntchito
Ikani phukusi la nodejs-stub-server pamtunda wa nginx rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverKuyesa Kupanikizika
Timayesa pogwiritsa ntchito benchmark ya Apache.
Ikani:
yum install -y httpd-toolsTimayamba kuyesa kugwiritsa ntchito benchmark ya Apache kuchokera ku ma seva 5 osiyanasiyana:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneKupanga Grafana
Simupeza dashboard patsamba lovomerezeka la Grafana.
Choncho, tidzachita ndi manja.
Mutha kupeza dashboard yanga yosungidwa .
Muyeneranso kupanga tebulo losinthika ndi zomwe zili mkati nginx.access_log.
Zofunsira za Singlestat:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Zopempha za Singlestat Zalephera:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Singlestat Failing Percent:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Nthawi Yoyankha ya Singlestat Avg:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Singlestat Max Yankho Nthawi:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Mawerengedwe Amtundu:
$columns(status, count(*) as c) from $table
Kuti mutulutse deta ngati chitumbuwa, muyenera kukhazikitsa pulogalamu yowonjezera ndikuyambitsanso grafana.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartPie TOP 5 Mkhalidwe:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
Komanso ndipereka zopempha popanda zowonera:
Werengani http_user_agent:
$columns(http_user_agent, count(*) c) FROM $tableGoodRate/BadRate:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tableNthawi Yoyankha:
$rate(avg(request_time) as request_time) FROM $tableNthawi yoyankha m'mwamba (nthawi yoyankhira m'mwamba):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableTable Count Status for all vhost:
$columns(status, count(*) as c) from $tableMawonekedwe onse a bolodi
Kuyerekeza kwa avg () ndi kuchuluka ()
pafupifupi ()
kuchuluka ()
Kutsiliza:
Ndikukhulupirira kuti anthu ammudzi atenga nawo mbali pakupanga / kuyesa ndikugwiritsa ntchito nginx-log-collector.
Ndipo wina akagwiritsa ntchito nginx-log-collector, amakuuzani kuchuluka komwe adasunga pa disk, RAM, ndi CPU.
Makanema a Telegraph:
Misekondi:
Kwa omwe ma milliseconds amafunikira, chonde lembani kapena kuvotani izi .
Source: www.habr.com