Ji ber ku bi zimanĂȘ Rust hatĂź nivĂźsandin, li gorĂź analogĂȘn xwe bi performansa bilind Ă» xerckirina kĂȘm RAM-ĂȘ tĂȘ destnĂźĆan kirin. Digel vĂȘ yekĂȘ, pir bal tĂȘ kiĆandin ser fonksiyonĂȘn ku bi rastbĂ»nĂȘ ve girĂȘdayĂź ne, nemaze, Ćiyana hilanĂźna bĂ»yerĂȘn neĆandiye li tamponek li ser dĂźskĂȘ Ă» zivirĂźna pelan.
Ji hĂȘla mĂźmarĂź ve, Vector routerek bĂ»yerĂȘ ye ku ji yek an jĂź bĂȘtir peyaman distĂźne çavkaniyĂȘn, vebijarkĂź li ser van peyaman sepandin veguherĂźnan, Ă» Ćandina wan ji yek an jĂź zĂȘdetir dirijĂźne.
Vektor ji bo pelĂȘbeat Ă» logstash-ĂȘ veguhezek e, ew dikare di her du rolan de tevbigere (qeydĂȘn wergirtin Ă» Ćandin), bĂȘtir hĂ»rguliyĂȘn li ser wan malperĂȘ.
Ger di Logstash de zincĂźre wekĂź tĂȘketin â ParzĂ»n â derketin çĂȘbibe wĂȘ hingĂȘ ew di Vector de ye çavkaniyĂȘn â veguherĂźne â diherikĂź
NimĂ»neyĂȘn di belgeyĂȘ de tĂȘne dĂźtin.
Ev talĂźmat talĂźmatek revĂźzekirĂź ye ji Vyacheslav Rakhinsky. RĂȘbernameyĂȘn orĂźjĂźnal pĂȘvajoyek geoip vedihewĂźne. Dema ceribandina geoip ji torgilokek navxweyĂź, vektor xeletiyek da.
Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=«geoip.country_name» rate_limit_secs=30
Ger kesek hewce bike ku geoip pĂȘvajoyĂȘ bike, wĂȘ hingĂȘ serĂź li rĂȘwerzĂȘn orĂźjĂźnal ĂȘn ji Vyacheslav Rakhinsky.
Em ĂȘ lihevhatina Nginx (qeydĂȘn gihĂźĆtinĂȘ) â Vektor (XerĂźdar | PelĂȘbeat) â Vektor (Server | Logstash) â veqetandĂź di Clickhouse de Ă» ji hev veqetandĂź di Elasticsearch de mĂźheng bikin. Em ĂȘ 4 pĂȘĆkĂȘĆkeran saz bikin. Her çend hĂ»n dikarin wĂȘ bi 3 serveran re derbas bikin.
Plan tiĆtekĂź bi vĂź rengĂź ye.
Selinux li ser hemĂź serverĂȘn xwe neçalak bikin
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
reboot
Em li ser hemß serveran emulatorek servera HTTP + karûbar saz dikin
ClickHouse koma rĂȘwerzĂȘn SSE 4.2 bikar tĂźne, ji ber vĂȘ yekĂȘ heya ku wekĂź din neyĂȘ destnĂźĆan kirin, piĆtgirĂźya wĂȘ di pĂȘvajoya ku hatĂź bikar anĂźn de dibe pĂȘdivĂźyek pergalĂȘ. Li vir emrĂȘ ye ku hĂ»n kontrol bikin ka pĂȘvajoya heyĂź SSE 4.2 piĆtgirĂź dike:
Veavakirina Elasticsearch-ĂȘ ji bo moda yek-girĂȘk 1 ĆirĂźn, 0 replica. Bi ĂźhtĂźmaleke mezin hĂ»n ĂȘ komek ji hejmareke mezin pĂȘĆkĂȘĆkeran hebin Ă» hĂ»n ne hewce ne ku vĂȘ yekĂȘ bikin.
Ji bo navnĂźĆĂȘn pĂȘĆerojĂȘ, ĆablonĂȘ xwerĂ» nĂ»ve bikin:
PiĆtĂź çĂȘkirina tabloyan, hĂ»n dikarin Vector bimeĆĂźnin
systemctl enable vector
systemctl start vector
TĂȘketinĂȘn vektorĂź dikarin bi vĂź rengĂź werin dĂźtin:
journalctl -f -u vector
DivĂȘ navnĂźĆĂȘn bi vĂź rengĂź di qeydan de hebin
INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.
Li ser xerĂźdar (pĂȘĆkĂȘĆkara malperĂȘ) - pĂȘĆkĂȘĆkara 1
Li ser servera bi nginx, hĂ»n hewce ne ku ipv6 neçalak bikin, ji ber ku tabloya tĂȘketinĂȘ li clickhouse zeviyĂȘ bikar tĂźne upstream_addr IPv4, ji ber ku ez ipv6 di hundurĂȘ torĂȘ de bikar naynim. Ger ipv6 neyĂȘ girtin, dĂȘ xeletĂź hebin:
DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)
Dibe ku xwendevanan, piĆtgiriya ipv6 zĂȘde bikin.
PĂȘĆĂź, pĂȘdivĂź ye ku em di pelĂȘ /etc/nginx/nginx.conf de formata tĂȘketinĂȘ ya Nginx-ĂȘ mĂźheng bikin.
user nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically
# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# provides the configuration file context in which the directives that affect connection processing are specified.
events {
# determines how much clients will be served per worker
# max clients = worker_connections * worker_processes
# max clients is also limited by the number of socket connections available on the system (~64k)
worker_connections 4000;
# optimized to serve many clients with each thread, essential for linux -- for testing environment
use epoll;
# accept as many connections as possible, may flood worker connections if set too low -- for testing environment
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format vector escape=json
'{'
'"node_name":"nginx-vector",'
'"timestamp":"$time_iso8601",'
'"server_name":"$server_name",'
'"request_full": "$request",'
'"request_user_agent":"$http_user_agent",'
'"request_http_host":"$http_host",'
'"request_uri":"$request_uri",'
'"request_scheme": "$scheme",'
'"request_method":"$request_method",'
'"request_length":"$request_length",'
'"request_time": "$request_time",'
'"request_referrer":"$http_referer",'
'"response_status": "$status",'
'"response_body_bytes_sent":"$body_bytes_sent",'
'"response_content_type":"$sent_http_content_type",'
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"remote_user": "$remote_user",'
'"upstream_addr": "$upstream_addr",'
'"upstream_bytes_received": "$upstream_bytes_received",'
'"upstream_bytes_sent": "$upstream_bytes_sent",'
'"upstream_cache_status":"$upstream_cache_status",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_header_time":"$upstream_header_time",'
'"upstream_response_length":"$upstream_response_length",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_status": "$upstream_status",'
'"upstream_content_type":"$upstream_http_content_type"'
'}';
access_log /var/log/nginx/access.log main;
access_log /var/log/nginx/access.json.log vector; # ĐĐŸĐČŃĐč Đ»ĐŸĐł ĐČ ŃĐŸŃĐŒĐ°ŃĐ” json
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
Ji bo ku hĂ»n veavakirina xweya heyĂź neĆikĂźnin, Nginx dihĂȘle hĂ»n çend rĂȘwerzĂȘn gihĂźĆtina_logĂȘ hebin
Ă di veavakirina /etc/vector/vector.toml de veguhertina Filebeat mĂźheng bikin. NavnĂźĆana IP-ĂȘ 172.26.10.108 navnĂźĆana IP-ya servera tĂȘketinĂȘ ye (Vector-Server)
data_dir = "/var/lib/vector"
[sources.nginx_file]
type = "file"
include = [ "/var/log/nginx/access.json.log" ]
start_at_beginning = false
fingerprinting.strategy = "device_and_inode"
[sinks.nginx_output_vector]
type = "vector"
inputs = [ "nginx_file" ]
address = "172.26.10.108:9876"
Ji bĂźr nekin ku bikarhĂȘnerĂȘ vektorĂȘ li koma hewce zĂȘde bikin da ku ew pelĂȘn tĂȘketinĂȘ bixwĂźne. MĂźnakĂź, nginx di centos de tĂȘketinĂȘn bi mafĂȘn koma adm diafirĂźne.
usermod -a -G adm vector
Werin em dest bi karĂ»barĂȘ vektorĂȘ bikin
systemctl enable vector
systemctl start vector
TĂȘketinĂȘn vektorĂź dikarin bi vĂź rengĂź werin dĂźtin:
journalctl -f -u vector
DivĂȘ navnĂźĆek bi vĂź rengĂź di qeydan de hebe
INFO vector::topology::builder: Healthcheck: Passed.
Testkirina StresĂȘ
Em ceribandinĂȘ bi karanĂźna pĂźvana Apache pĂȘk tĂźnin.
PakĂȘta httpd-tools li ser hemĂź pĂȘĆkĂȘĆkeran hate saz kirin
Em bi karanĂźna pĂźvana Apache ji 4 serverĂȘn cihĂȘreng ĂȘn li ser ekranĂȘ dest bi ceribandinĂȘ dikin. PĂȘĆĂźn, em multiplekserĂȘ termĂźnalĂȘ ekranĂȘ didin destpĂȘkirin, Ă» dĂ»v re jĂź em bi karanĂźna pĂźvana Apache dest bi ceribandinĂȘ dikin. Meriv çawa bi ekranĂȘ re dixebite ku hĂ»n dikarin tĂȘ de bibĂźnin gotara.
Ji servera 1st
while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; done
Ji servera 2st
while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; done
Ji servera 3st
while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; done
Ji servera 4st
while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; done
select concat(database, '.', table) as table,
formatReadableSize(sum(bytes)) as size,
sum(rows) as rows,
max(modification_time) as latest_modification,
sum(bytes) as bytes_size,
any(engine) as engine,
formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;
Ka em fĂȘr bibin ka çiqas tĂȘketin li Clickhouse girt.
Mezinahiya tabloya tĂȘketin 857.19 MB e.
Mezinahiya heman daneyĂȘ di navnĂźĆa Elasticsearch de 4,5 GB ye.
Heke hĂ»n di vektorĂȘ de di pĂźvanan de daneyan diyar nekin, Clickhouse 4500/857.19 = 5.24 carĂź ji Elasticsearch kĂȘmtir digire.
Di vektorĂȘ de, qada kompresyonĂȘ ji hĂȘla xwerĂ» ve tĂȘ bikar anĂźn.