Ka matapakihia e tenei tuhinga te kaupapa , ka panui i nga raarangi nginx, tukuna ki te roopu Clickhouse. Ko te tikanga ka whakamahia te ElasticSearch mo nga raarangi. Ko te Clickhouse he iti ake nga rauemi (mokowā kōpae, RAM, PTM). Ka tere ake te tuhi raraunga a Clickhouse. Ko te Clickhouse e kopiri ana i nga raraunga, na te mea ka pai ake nga raraunga o te kōpae. Ka kitea nga painga o Clickhouse i roto i nga kiriata 2 mai i te ripoata
Hei tiro i nga tātaritanga ma nga raarangi, me hanga he papatohu mo Grafana.
Who cares, welcome under cat.
Tāuta nginx, grafana i te ara paerewa.
Tāuta kāhui clickhouse me te pukapuka-pukapuka mai .
Te hanga i te papaa raraunga me nga ripanga i Clickhouse
I tenei Ko nga patai a SQL mo te hanga papaaarangi me nga ripanga mo te nginx-log-collector i Clickhouse e whakaahuatia ana.
Ka tukuna e matou ia tono i runga i ia tūmau o te kohinga Clickhouse.
Tuhipoka nui. I tenei rarangi, me whakakapia te logs_cluster ki to ingoa tautau mai i te konae clickhouse_remote_servers.xml i waenganui i "mamao_servers" me "shard".
ENGINE = Distributed('logs_cluster', 'nginx', 'access_log_shard', rand())Te whakauru me te whirihora nginx-log-collector-rpm
Karekau he rpm a Nginx-log-collector. I konei hanga rpm mo taua mea. ka hangaia te rpm ma te whakamahi
Tāutahia te mōkihi rpm nginx-log-collector-rpm
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nginx-log-collector-rpm
yum -y install nginx-log-collector
systemctl start nginx-log-collectorWhakatika whirihora /etc/nginx-log-collector/config.yaml:
.......
upload:
table: nginx.access_log
dsn: http://ip-адрес-кластера-clickhouse:8123/
- tag: "nginx_error:"
format: error # access | error
buffer_size: 1048576
upload:
table: nginx.error_log
dsn: http://ip-адрес-кластера-clickhouse:8123/Te whakatu nginx
Whirihoranga nginx whānui:
user nginx;
worker_processes auto;
#error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format avito_json escape=json
'{'
'"event_datetime": "$time_iso8601", '
'"server_name": "$server_name", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"http_x_real_ip": "$http_x_real_ip", '
'"status": "$status", '
'"scheme": "$scheme", '
'"request_method": "$request_method", '
'"request_uri": "$request_uri", '
'"server_protocol": "$server_protocol", '
'"body_bytes_sent": $body_bytes_sent, '
'"http_referer": "$http_referer", '
'"http_user_agent": "$http_user_agent", '
'"request_bytes": "$request_length", '
'"request_time": "$request_time", '
'"upstream_addr": "$upstream_addr", '
'"upstream_response_time": "$upstream_response_time", '
'"hostname": "$hostname", '
'"host": "$host"'
'}';
access_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx avito_json; #ClickHouse
error_log syslog_server=unix:/var/run/nginx_log.sock,nohostname,tag=nginx_error; #ClickHouse
#access_log /var/log/nginx/access.log main;
proxy_ignore_client_abort on;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Kaihautū mariko tetahi:
vhost1.conf:
upstream backend {
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
server ip-адрес-сервера-с-stub_http_server:8080;
}
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://backend;
}
}Tāpirihia nga kaihautu mariko ki te kōnae /etc/hosts:
ip-адрес-сервера-с-nginx vhost1Emulator tūmau HTTP
Hei emulator tūmau HTTP ka whakamahia e matou i
nodejs-stub-server kaore he rpm. I konei hanga rpm mo taua mea. ka hangaia te rpm ma te whakamahi
Tāutahia te mōkihi nodejs-stub-server ki te rpm nginx whakarunga
yum -y install yum-plugin-copr
yum copr enable antonpatsev/nodejs-stub-server
yum -y install stub_http_server
systemctl start stub_http_serverWhakamātautau ahotea
Ka mahia nga whakamatautau ma te whakamahi i te tohu tohu a Apache.
Tāutahia:
yum install -y httpd-toolsKa timata taatau ki te whakamatautau ma te whakamahi i te tohu tohu Apache mai i nga kaitoro rereke e 5:
while true; do ab -H "User-Agent: 1server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 2server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 3server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 4server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; done
while true; do ab -H "User-Agent: 5server" -c 10 -n 10 -t 10 http://vhost1/; sleep 1; doneTe whakatu i te Grafana
Kaore koe e kitea he papatohu i runga i te paetukutuku mana o Grafana.
No reira, ka mahia ma te ringaringa.
Ka kitea e koe taku papatohu kua tiakina .
Me hanga ano he taurangi ripanga me nga ihirangi nginx.access_log.
Singlestat Tapeke Tono:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter GROUP BY t
Nga Tono I Rahua:
SELECT
1 as t,
count(*) as c
FROM $table
WHERE $timeFilter AND status NOT IN (200, 201, 401) GROUP BY t
Takitahi Rahua Ōrau:
SELECT
1 as t, (sum(status = 500 or status = 499)/sum(status = 200 or status = 201 or status = 401))*100 FROM $table
WHERE $timeFilter GROUP BY t
Singlestat Avg Wā Whakautu:
SELECT
1, avg(request_time) FROM $table
WHERE $timeFilter GROUP BY 1
Singlestat Max Wā Whakautu:
SELECT
1 as t, max(request_time) as c
FROM $table
WHERE $timeFilter GROUP BY t
Tūnga Tatau:
$columns(status, count(*) as c) from $table
Hei whakaputa i nga raraunga penei i te porowhita, me whakauru koe i te mono me te uta ano i te grafana.
grafana-cli plugins install grafana-piechart-panel
service grafana-server restartPorohita TOP 5 Tūnga:
SELECT
1, /* fake timestamp value */
status,
sum(status) AS Reqs
FROM $table
WHERE $timeFilter
GROUP BY status
ORDER BY Reqs desc
LIMIT 5
I tua atu ka hoatu e ahau nga tono kaore he whakaahua:
Tatau http_kaiwhakamahi_kaiwhakamahi:
$columns(http_user_agent, count(*) c) FROM $tableReanga Pai/Kino:
$rate(countIf(status = 200) AS good, countIf(status != 200) AS bad) FROM $tablewā whakautu:
$rate(avg(request_time) as request_time) FROM $tableWā whakautu whakarunga (wa whakautu o te 1st whakarunga):
$rate(avg(arrayElement(upstream_response_time,1)) as upstream_response_time) FROM $tableTūnga Tatau Ripanga mō ngā vhosts katoa:
$columns(status, count(*) as c) from $tableTirohanga whanui mo te papatohu
Te whakatairite toharite() me te rahi()
toharite()
rahi()
Whakamutunga:
Ko te tumanako ka uru te hapori ki te whakawhanake/whakamatautau me te whakamahi i te nginx-log-collector.
A, ka whakatinanahia e tetahi te nginx-log-collector, ka korerotia e ia ki a koe te nui o tana penapena kōpae, RAM, PTM.
Nga hongere Telegram:
Milihekona:
Ko wai e whakaaro nui ana ki nga manomano, tuhia, pooti ranei, koa, i roto i tenei .
Source: will.com