, ืืืืืื ื ืฆื ืืึทืืืขื, ืืืขืจืืึทืื ืืื ืฉืืงื ืงืืึธืฅ ืืึทืื, ืืขืืจืืงืก ืืื ืืขืฉืขืขื ืืฉื.
โ
ืขืก ืืื ืืขืฉืจืืื ืืื ืื ืจืืกื ืฉืคึผืจืึทื, ืขืก ืืื ืงืขืจืึทืงืืขืจืืืื ืืืจื ืืืื ืคืึธืจืฉืืขืืื ื ืืื ื ืืืขืจืืง ืืึทืจืึทื ืงืึทื ืกืึทืืฉืึทื ืงืึทืืคึผืขืจื ืืื ืืืึทื ืึทื ืึทืืึธืืืขืก. ืืื ืึทืืืฉืึทื, ืคืื ืืคืืขืจืงืืึทืืงืืื ืืื ืืึทืฆืึธืื ืฆื ืคืึทื ืืงืฉืึทื ื ืฉืืึทืืืช ืฆื ืงืขืจืขืงืื ืึทืก, ืกืคึผืขืฆืืขื ืื ืคืืืืงืืื ืฆื ืจืึทืืขืืืขื ืึทื ืกืขื ื ืืขืฉืขืขื ืืฉื ืฆื ืึท ืืึทืคืขืจ ืืืืฃ ืืืกืง ืืื ืืจืืืขื ืืขืงืขืก.
ืึทืจืงืึทืืขืงืืฉืขืจืึทืื, ืืืขืงืืึธืจ ืืื ืึท ืืขืฉืขืขื ืืฉ ืจืึทืืืขืจ ืืืึธืก ื ืขืื ืึทืจืืืงืืขื ืคืื ืืืื ืขืจ ืึธืืขืจ ืืขืจ ืงืืืขืืขืจ, ืึทืคึผืฉืึทื ืึทื ืึทืคึผืืืืื ื ืืืืขืจ ืื ืึทืจืืืงืืขื ืืจืึทื ืกืคืึธืจืืึทืฆืืขืก, ืืื ืฉืืงื ืืื ืฆื ืืืื ืขืจ ืึธืืขืจ ืืขืจ ืืจืืื ื.
ืืืขืงืืึธืจ ืืื ืึท ืคืึทืจืืืึทื ืคึฟืึทืจ Filebeat ืืื Logstash, ืขืก ืงืขื ืขื ืฉืคึผืืื ืืื ืืืืืข ืจืึธืืขืก (ืืึทืงืืืขื ืืื ืฉืืงื ืืึธืืก), ืืขืจ ืืขืืึทืืืก ืืืืฃ ืืื .
ืืืื ืืื Logstash ืื ืงืืื ืืื ืืขืืืื ืืื ืึทืจืืึทื ืฉืจืืึทื โ ืคืืืืขืจ โ ืจืขืืืืืึทื, ืืื ืืืขืงืืึธืจ ืขืก ืืื โ โ
ืืืืฉืคืืื ืงืขื ืขื ืืืื ืืขืคึฟืื ืขื ืืื ืื ืืึทืงืืืืขื ืืืืฉืึทื.
ืื ืืืืขื ืืื ืึท ืจืืืืืืื ืืืืขื ืคืื . ืืขืจ ืึธืจืืืื ืขื ืื ืกืืจืึทืงืฉืึทื ื ืึทื ืืืึทืืื ืืขืึธืืคึผ ืคึผืจืึทืกืขืกืื ื. ืืืขื ืืขืกืืื ื ืืขืึธืืคึผ ืคึฟืื ืึทื ืื ืขืจืืขื ื ืขืฅ, ืืืขืงืืึธืจ ืืื ืึท ืืขืืช.
Aug 05 06:25:31.889 DEBUG transform{name=nginx_parse_rename_fields type=rename_fields}: vector::transforms::rename_fields: Field did not exist field=ยซgeoip.country_nameยป rate_limit_secs=30ืืืื ืืืขืจ ืขืก ืื ืืึทืจืฃ ืฆื ืคึผืจืึธืฆืขืก ืืขืึธืืคึผ, ืึธืคึผืฉืืงื ืฆื ืื ืึธืจืืืื ืขื ืื ืกืืจืึทืงืฉืึทื ื ืคืื .
ืืืจ ืืืขืื ืงืึทื ืคืืืืขืจ ืื ืงืึธืืืื ืึทืฆืืข ืคืื โโโโื ืืื ืงืก (ืึทืงืกืขืก ืืึธืืก) โ ืืืขืงืืึธืจ (ืงืืืขื ื | ืคืืืขืืขืึทื) โ ืืืขืงืืึธืจ (ืกืขืจืืืืจืขืจ | ืืึธืืกืืึทืฉ) โ ืกืขืคึผืขืจืึทืืื ืืื ืงืืืงืงืืึธืืกืข ืืื ืกืขืคึผืขืจืึทืืื ืืื Elasticsearch. ืืืจ ืืืขืื ืื ืกืืึทืืืจื 4 ืกืขืจืืืขืจืก. ืืึธืืฉ ืืืจ ืงืขื ืขื ืืืืคึผืึทืก ืขืก ืืื 3 ืกืขืจืืืขืจืก.
ืืขืจ ืกืืขืืข ืืื ืขืคึผืขืก ืืื ืืึธืก.
ืืืกืืืืึทื ืกืขืืื ืืงืก ืืืืฃ ืึทืืข ืืืื ืกืขืจืืืขืจืก
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
rebootืืืจ ืื ืกืืึทืืืจื ืึท ืืืืคึผ ืกืขืจืืืขืจ ืขืืืืึทืืึธืจ + ืืืืืืึทืืื ืืืืฃ ืึทืืข ืกืขืจืืืขืจืก
ืืื ืึท ืืืืคึผ ืกืขืจืืืขืจ ืขืืืืึทืืึธืจ ืืืจ ืืืขืื ื ืืฆื ืคืื
Nodejs-stub-server ืืื ื ืืฉื ืึท ืจืคึผื. ืฉืึทืคึฟื ืจืคึผื ืคึฟืึทืจ ืขืก. rpm ืืืขื ืืืื ืงืึทืืคึผืืืื ืืื
ืืืื ืื antonpatsev/nodejs-stub-server ืจืืคึผืึทืืึทืืึธืจื
yum -y install yum-plugin-copr epel-release
yes | yum copr enable antonpatsev/nodejs-stub-serverืื ืกืืึทืืืจื nodejs-stub-server, Apache ืืขื ืืฉืืึทืจืง ืืื ืคืึทืจืฉืืขืื ืืืึธืงืืึทื ืืืืืืคึผืืขืงืกืขืจ ืืืืฃ ืึทืืข ืกืขืจืืืขืจืก
yum -y install stub_http_server screen mc httpd-tools screenืืื ืงืขืจืขืงืืึทื ืื stub_http_server ืขื ืืคืขืจ ืฆืืื ืืื ืื /var/lib/stub_http_server/stub_http_server.js ืืขืงืข ืึทืืื ืึทื ืขืก ืืขื ืขื ืืขืืืขื ืืขืจ ืืึธืืก.
var max_sleep = 10;ืืื ืก ืงืึทืืขืจ stub_http_server.
systemctl start stub_http_server
systemctl enable stub_http_serverืืืืฃ ืกืขืจืืืขืจ 3
ClickHouse ื ืืฆื ืื SSE 4.2 ืื ืกืืจืืงืืืึธื ืก ืฉืืขืื, ืึทืืื ืืืื ื ืื ืึทื ืืขืจืฉ ืกืคึผืขืกืืคืืขื, ืฉืืืฆื ืคึฟืึทืจ ืขืก ืืื ืื ืคึผืจืึทืกืขืกืขืจ ืืขืืืืื ื ืืืขืจื ืึทื ื ืึธื ืกืืกืืขื ืคืึธืืขืจืื ื. ืืึธ ืืื ืื ืืึทืคึฟืขื ืฆื ืงืึธื ืืจืึธืืืจื ืืืื ืื ืงืจืึทื ื ืคึผืจืึทืกืขืกืขืจ ืฉืืืฆื SSE 4.2:
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"ืขืจืฉืืขืจ ืืืจ ืืึทืจืคึฟื ืฆื ืคืึทืจืืื ืื ืื ืืึทืึทืืืขืจ ืจืืคึผืึทืืึทืืึธืจื:
sudo yum install -y yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64ืฆื ืื ืกืืึทืืืจื ืคึผืึทืงืึทืืืฉืึทื ืืืจ ืืึทืจืคึฟื ืฆื ืืืืคื ืื ืคืืืืขื ืืข ืงืึทืืึทื ืื:
sudo yum install -y clickhouse-server clickhouse-clientืืึธืื ืงืืืงืืึธืืกืข-ืกืขืจืืืขืจ ืฆื ืืขืจื ืฆื ืื ื ืขืฅ ืงืึธืจื ืืื ืืขืจ ืืขืงืข /etc/clickhouse-server/config.xml
<listen_host>0.0.0.0</listen_host>ืืฉืึทื ืืื ื ืื ืืึธืืื ื ืืืจืื ืคืื ืฉืคึผืืจ ืฆื ืืืืึทื
ืืขืืื
ื ืึธืจืืึทื ืงืึทืืคึผืจืขืฉืึทื ืกืขืืืื ืืก:
min_compress_block_size 65536
max_compress_block_size 1048576ืฆื ืึทืงืืึทืืืืื Zstd ืงืึทืืคึผืจืขืฉืึทื, ืขืก ืืื ืืขืืืขื ืึทืืืืืืื ื ืืฉื ืฆื ืคืึทืจืืื ืื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื, ืึธืืขืจ ืฆื ื ืืฆื DDL.
ืืื ืงืขื ื ืืฉื ืืขืคึฟืื ืขื ืืื ืฆื ื ืืฆื zstd ืงืึทืืคึผืจืขืฉืึทื ืืืจื DDL ืืื Google. ืึทืืื ืืื ืืื ืงืก ืขืก ืืื ืืื.
ืืืจืื ืืืึธืก ื ืืฆื zstd ืงืึทืืคึผืจืขืฉืึทื ืืื Clickhouse, ืืืืข ืืืืื ืื ืื ืกืืจืึทืงืฉืึทื ื.
ืฆื ืึธื ืืืืื ืืขื ืกืขืจืืืขืจ ืืื ืึท ืืืืืึทื, ืืืืคื:
service clickhouse-server startืืืฆื ืืึธืื ืก ืืึทื ืืืืฃ ืฆื ืืึทืฉืืขืืืงื Clickhouse
ืืืื ืฆื Clickhouse
clickhouse-client -h 172.26.10.109 -m172.26.10.109 - IP ืคืื ืื ืกืขืจืืืขืจ ืืื ืงืืืงืืึธืืกืข ืืื ืืื ืกืืึทืืืจื.
ืืื ืก ืฉืึทืคึฟื ืึท ืืืขืงืืึธืจ ืืึทืืึทืืืืก
CREATE DATABASE vector;ืืื ืก ืงืึธื ืืจืึธืืืจื ืึทื ืื ืืึทืืึทืืืืก ืืืืืกืฅ.
show databases;ืฉืึทืคึฟื ืึท ืืืขืงืืึธืจ.ืืึธืืก ืืืฉ.
/* ะญัะพ ัะฐะฑะปะธัะฐ ะณะดะต ั
ัะฐะฝัััั ะปะพะณะธ ะบะฐะบ ะตััั */
CREATE TABLE vector.logs
(
`node_name` String,
`timestamp` DateTime,
`server_name` String,
`user_id` String,
`request_full` String,
`request_user_agent` String,
`request_http_host` String,
`request_uri` String,
`request_scheme` String,
`request_method` String,
`request_length` UInt64,
`request_time` Float32,
`request_referrer` String,
`response_status` UInt16,
`response_body_bytes_sent` UInt64,
`response_content_type` String,
`remote_addr` IPv4,
`remote_port` UInt32,
`remote_user` String,
`upstream_addr` IPv4,
`upstream_port` UInt32,
`upstream_bytes_received` UInt64,
`upstream_bytes_sent` UInt64,
`upstream_cache_status` String,
`upstream_connect_time` Float32,
`upstream_header_time` Float32,
`upstream_response_length` UInt64,
`upstream_response_time` Float32,
`upstream_status` UInt16,
`upstream_content_type` String,
INDEX idx_http_host request_http_host TYPE set(0) GRANULARITY 1
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
TTL timestamp + toIntervalMonth(1)
SETTINGS index_granularity = 8192;ืืืจ ืงืึธื ืืจืึธืืืจื ืึทื ืื ืืืฉื ืืขื ืขื ืืืฉืืคื. ืืื ืก ืงืึทืืขืจ clickhouse-client ืืื ืืึทืื ืึท ืืงืฉื.
ืืื ืก ืืืื ืฆื ืื ืืืขืงืืึธืจ ืืึทืืึทืืืืก.
use vector;
Ok.
0 rows in set. Elapsed: 0.001 sec.ืืึธืืืจ ืงืืงื ืืืืฃ ืื ืืืฉื.
show tables;
โโnameโโโโโโโโโโโโโโโโโ
โ logs โ
โโโโโโโโโโโโโโโโโโโโโโโืื ืกืืึธืืื ื Elasticsearch ืืืืฃ ืื 4 ืกืขืจืืืขืจ ืฆื ืฉืืงื ืื ืืขืืืข ืืึทืื ืฆื Elasticsearch ืคึฟืึทืจ ืคืึทืจืืืืึทื ืืื Clickhouse
ืืืื ืึท ืฆืืืืจ ืจืคึผื ืฉืืืกื
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchืืึธืืืจ ืืึทืื 2 ืจืขืคึผืึธ:
/etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md/etc/yum.repos.d/kibana.repo
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-mdืื ืกืืึทืืืจื Elasticsearch ืืื Kibana
yum install -y kibana elasticsearchืืื ื ืขืก ืืืขื ืืืื ืืื 1 ืงืึธืคึผืืข, ืืืจ ืืึทืจืคึฟื ืฆื ืืืืื ืื ืคืืืืขื ืืข ืฆื ืื /etc/elasticsearch/elasticsearch.yml ืืขืงืข:
discovery.type: single-nodeืึทืืื ืึทื ืืืขืงืืึธืจ ืงืขื ืขื ืฉืืงื ืืึทืื ืฆื Elasticsearch ืคึฟืื ืื ืื ืืขืจ ืกืขืจืืืขืจ, ืืึธืื ืืื ืื ืืืืฉื network.host.
network.host: 0.0.0.0ืฆื ืคืึทืจืืื ืื ืฆื kibana, ืืืืฉื ืื ืกืขืจืืืขืจ.ืืึธืกื ืคึผืึทืจืึทืืขืืขืจ ืืื ืืขืจ ืืขืงืข /etc/kibana/kibana.yml
server.host: "0.0.0.0"ืึทืื ืืื ืึทืจืืึทื ื ืขืืขื Elasticsearch ืืื ืึทืืืึธืกืืึทืจื
systemctl enable elasticsearch
systemctl start elasticsearchืืื ืงืืืึทื ืึท
systemctl enable kibana
systemctl start kibanaืงืึทื ืคืืืืขืจ Elasticsearch ืคึฟืึทืจ ืืืื-ื ืึธืืข ืืึธืืข 1 ืฉืึทืจื, 0 ืจืขืคึผืืืงืข. ืจืืึฟ ืืกืชึผืื ืืืจ ืืืขื ืืึธืื ืึท ืงื ืืื ืคืื ืึท ืืจืืืก ื ืืืขืจ ืคืื ืกืขืจืืืขืจืก ืืื ืืืจ ืืึธื ื ืื ืืึทืจืคึฟื ืฆื ืืึธื ืืึธืก.
ืคึฟืึทืจ ืฆืืงืื ืคึฟื ืื ืืขืงืกืื, ืืขืจืืืึทื ืืืงื ืื ืคืขืืืงืืึทื ืืืกืืขืจ:
curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}' ืื ืกืืึทืืืจืื ื ืืื ืึท ืคืึทืจืืืึทื ืคึฟืึทืจ Logstash ืืืืฃ ืกืขืจืืืขืจ 2
yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpm mc httpd-tools screenืืึธืืืจ ืฉืืขืื ืืืขืงืืึธืจ ืืื ืึท ืคืึทืจืืืึทื ืคึฟืึทืจ Logstash. ืขืืืืื ื ืื ืืขืงืข /etc/vector/vector.toml
# /etc/vector/vector.toml
data_dir = "/var/lib/vector"
[sources.nginx_input_vector]
# General
type = "vector"
address = "0.0.0.0:9876"
shutdown_timeout_secs = 30
[transforms.nginx_parse_json]
inputs = [ "nginx_input_vector" ]
type = "json_parser"
[transforms.nginx_parse_add_defaults]
inputs = [ "nginx_parse_json" ]
type = "lua"
version = "2"
hooks.process = """
function (event, emit)
function split_first(s, delimiter)
result = {};
for match in (s..delimiter):gmatch("(.-)"..delimiter) do
table.insert(result, match);
end
return result[1];
end
function split_last(s, delimiter)
result = {};
for match in (s..delimiter):gmatch("(.-)"..delimiter) do
table.insert(result, match);
end
return result[#result];
end
event.log.upstream_addr = split_first(split_last(event.log.upstream_addr, ', '), ':')
event.log.upstream_bytes_received = split_last(event.log.upstream_bytes_received, ', ')
event.log.upstream_bytes_sent = split_last(event.log.upstream_bytes_sent, ', ')
event.log.upstream_connect_time = split_last(event.log.upstream_connect_time, ', ')
event.log.upstream_header_time = split_last(event.log.upstream_header_time, ', ')
event.log.upstream_response_length = split_last(event.log.upstream_response_length, ', ')
event.log.upstream_response_time = split_last(event.log.upstream_response_time, ', ')
event.log.upstream_status = split_last(event.log.upstream_status, ', ')
if event.log.upstream_addr == "" then
event.log.upstream_addr = "127.0.0.1"
end
if (event.log.upstream_bytes_received == "-" or event.log.upstream_bytes_received == "") then
event.log.upstream_bytes_received = "0"
end
if (event.log.upstream_bytes_sent == "-" or event.log.upstream_bytes_sent == "") then
event.log.upstream_bytes_sent = "0"
end
if event.log.upstream_cache_status == "" then
event.log.upstream_cache_status = "DISABLED"
end
if (event.log.upstream_connect_time == "-" or event.log.upstream_connect_time == "") then
event.log.upstream_connect_time = "0"
end
if (event.log.upstream_header_time == "-" or event.log.upstream_header_time == "") then
event.log.upstream_header_time = "0"
end
if (event.log.upstream_response_length == "-" or event.log.upstream_response_length == "") then
event.log.upstream_response_length = "0"
end
if (event.log.upstream_response_time == "-" or event.log.upstream_response_time == "") then
event.log.upstream_response_time = "0"
end
if (event.log.upstream_status == "-" or event.log.upstream_status == "") then
event.log.upstream_status = "0"
end
emit(event)
end
"""
[transforms.nginx_parse_remove_fields]
inputs = [ "nginx_parse_add_defaults" ]
type = "remove_fields"
fields = ["data", "file", "host", "source_type"]
[transforms.nginx_parse_coercer]
type = "coercer"
inputs = ["nginx_parse_remove_fields"]
types.request_length = "int"
types.request_time = "float"
types.response_status = "int"
types.response_body_bytes_sent = "int"
types.remote_port = "int"
types.upstream_bytes_received = "int"
types.upstream_bytes_send = "int"
types.upstream_connect_time = "float"
types.upstream_header_time = "float"
types.upstream_response_length = "int"
types.upstream_response_time = "float"
types.upstream_status = "int"
types.timestamp = "timestamp"
[sinks.nginx_output_clickhouse]
inputs = ["nginx_parse_coercer"]
type = "clickhouse"
database = "vector"
healthcheck = true
host = "http://172.26.10.109:8123" # ะะดัะตั Clickhouse
table = "logs"
encoding.timestamp_format = "unix"
buffer.type = "disk"
buffer.max_size = 104900000
buffer.when_full = "block"
request.in_flight_limit = 20
[sinks.elasticsearch]
type = "elasticsearch"
inputs = ["nginx_parse_coercer"]
compression = "none"
healthcheck = true
# 172.26.10.116 - ัะตัะฒะตั ะณะดะต ัััะฐะฝะพะฒะตะฝ elasticsearch
host = "http://172.26.10.116:9200"
index = "vector-%Y-%m-%d"ืืืจ ืงืขื ืขื ืกืืจืืืขืจื ืื Transforms.nginx_parse_add_defaults ืึธืคึผืืืืืื ื.
ืืื ื ืืฆื ืื ืงืึธื ืคืืืก ืคึฟืึทืจ ืึท ืงืืืื CDN ืืื ืขืก ืงืขื ืขื ืืืื ืขืืืขืืข ืืืึทืืืขืก ืืื ืึทืคึผืกืืจืื_*
ืคึฟืึทืจ ืืืึทืฉืคึผืื:
"upstream_addr": "128.66.0.10:443, 128.66.0.11:443, 128.66.0.12:443"
"upstream_bytes_received": "-, -, 123"
"upstream_status": "502, 502, 200"ืืืื ืืึธืก ืืื ื ืืฉื ืืืื ืกืืืืึทืฆืืข, ืืขื ืึธืคึผืืืืืื ื ืงืขื ืขื ืืืื ืกืืืคึผืืึทืคืืื
ืืึธืืืจ ืืึทืื ืกืขืจืืืืก ืกืขืืืื ืืก ืคึฟืึทืจ systemd /etc/systemd/system/vector.service
# /etc/systemd/system/vector.service
[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target
[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector
[Install]
WantedBy=multi-user.targetื ืึธื ืงืจืืืืืื ื ืื ืืืฉื, ืืืจ ืงืขื ืขื ืืืืคื ืืืขืงืืึธืจ
systemctl enable vector
systemctl start vectorืืืขืงืืึธืจ ืืึธืืก ืงืขื ืขื ืืืื ืืืืื ืืื ืืึธืก:
journalctl -f -u vectorืขืก ืืึธื ืืืื ืืืื ืกื ืืื ืืึธืก ืืื ืื ืืึธืืก
INFO vector::topology::builder: Healthcheck: Passed.
INFO vector::topology::builder: Healthcheck: Passed.ืืืืฃ ืืขื ืงืืืขื ื (ืืืขื ืกืขืจืืืขืจ) - 1 ืกืขืจืืืขืจ
ืืืืฃ ืื ืกืขืจืืืขืจ ืืื nginx, ืืืจ ืืึทืจืคึฟื ืฆื ืืืกืืืืึทื ืืคึผืื6, ืืืืึทื ืื ืืึธืืก ืืืฉ ืืื ืงืืืงืืึทืืก ื ืืฆื ืืขื ืคืขืื upstream_addr IPv4, ืืืืึทื ืืื ืืึธื ื ืื ื ืืฆื ืืคึผืื6 ืืื ืื ื ืขืฅ. ืืืื ipv6 ืืื ื ืืฉื ืืืืกืืขืืจืืื ืึทืืืขืง, ืขืก ืืืขื ืืืื ืขืจืจืึธืจืก:
DB::Exception: Invalid IPv4 value.: (while read the value of key upstream_addr)ืืึธืืขืจ ืืืืขื ืขืจ, ืืืืื ืืคึผืื6 ืฉืืืฆื.
ืฉืึทืคึฟื ืื ืืขืงืข /etc/sysctl.d/98-disable-ipv6.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1ืึทืคึผืืืืื ื ืื ืกืขืืืื ืืก
sysctl --systemืืึธืืืจ ืื ืกืืึทืืืจื nginx.
ืฆืืืขืืืืื ื ืืื ืงืก ืจืืคึผืึทืืึทืืึธืจื ืืขืงืข /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=trueืื ืกืืึทืืืจื ืื nginx ืคึผืขืงื
yum install -y nginxืขืจืฉืืขืจ, ืืืจ ืืึทืจืคึฟื ืฆื ืงืึทื ืคืืืืขืจ ืื ืงืืึธืฅ ืคึฟืึธืจืืึทื ืืื Nginx ืืื ืืขืจ ืืขืงืข /etc/nginx/nginx.conf
user nginx;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically
# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# provides the configuration file context in which the directives that affect connection processing are specified.
events {
# determines how much clients will be served per worker
# max clients = worker_connections * worker_processes
# max clients is also limited by the number of socket connections available on the system (~64k)
worker_connections 4000;
# optimized to serve many clients with each thread, essential for linux -- for testing environment
use epoll;
# accept as many connections as possible, may flood worker connections if set too low -- for testing environment
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format vector escape=json
'{'
'"node_name":"nginx-vector",'
'"timestamp":"$time_iso8601",'
'"server_name":"$server_name",'
'"request_full": "$request",'
'"request_user_agent":"$http_user_agent",'
'"request_http_host":"$http_host",'
'"request_uri":"$request_uri",'
'"request_scheme": "$scheme",'
'"request_method":"$request_method",'
'"request_length":"$request_length",'
'"request_time": "$request_time",'
'"request_referrer":"$http_referer",'
'"response_status": "$status",'
'"response_body_bytes_sent":"$body_bytes_sent",'
'"response_content_type":"$sent_http_content_type",'
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"remote_user": "$remote_user",'
'"upstream_addr": "$upstream_addr",'
'"upstream_bytes_received": "$upstream_bytes_received",'
'"upstream_bytes_sent": "$upstream_bytes_sent",'
'"upstream_cache_status":"$upstream_cache_status",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_header_time":"$upstream_header_time",'
'"upstream_response_length":"$upstream_response_length",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_status": "$upstream_status",'
'"upstream_content_type":"$upstream_http_content_type"'
'}';
access_log /var/log/nginx/access.log main;
access_log /var/log/nginx/access.json.log vector; # ะะพะฒัะน ะปะพะณ ะฒ ัะพัะผะฐัะต json
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}ืืื ืกืืจ ื ืืฉื ืฆื ืืจืขืื ืืืื ืงืจืึทื ื ืงืึทื ืคืืืืขืจืืืฉืึทื, Nginx ืึทืืึทืื ืืืจ ืฆื ืืึธืื ืขืืืขืืข ืึทืงืกืขืก_ืืึธื ืืืืจืขืงืืืืื
access_log /var/log/nginx/access.log main; # ะกัะฐะฝะดะฐััะฝัะน ะปะพะณ
access_log /var/log/nginx/access.json.log vector; # ะะพะฒัะน ะปะพะณ ะฒ ัะพัะผะฐัะต jsonืื ืืืืกื ื ืืฉื ืคืึทืจืืขืกื ืฆื ืืืืื ืึท ืืขืจืฉื ืฆื ืืึธืืจืึธืืึทืืข ืคึฟืึทืจ ื ืืึทืข ืืึธืืก (ืืืื ืื ืืึธื ืืขืงืข ืืื ื ืืฉื ืกืืฃ ืืื .ืืึธื)
ืึทืจืึธืคึผื ืขืืขื default.conf ืคืื /etc/nginx/conf.d/
rm -f /etc/nginx/conf.d/default.confืืืื ืืืืจืืืขื ืืึทืืขืืึธืก /etc/nginx/conf.d/vhost1.conf
server {
listen 80;
server_name vhost1;
location / {
proxy_pass http://172.26.10.106:8080;
}
}ืืืื ืืืืจืืืขื ืืึทืืขืืึธืก /etc/nginx/conf.d/vhost2.conf
server {
listen 80;
server_name vhost2;
location / {
proxy_pass http://172.26.10.108:8080;
}
}ืืืื ืืืืจืืืขื ืืึทืืขืืึธืก /etc/nginx/conf.d/vhost3.conf
server {
listen 80;
server_name vhost3;
location / {
proxy_pass http://172.26.10.109:8080;
}
}ืืืื ืืืืจืืืขื ืืึทืืขืืึธืก /etc/nginx/conf.d/vhost4.conf
server {
listen 80;
server_name vhost4;
location / {
proxy_pass http://172.26.10.116:8080;
}
}ืืืื ืืืืจืืืึทื ืืื ืืช (172.26.10.106 ืืคึผ ืคืื ืื ืกืขืจืืืขืจ ืืื nginx ืืื ืืื ืกืืึทืืืจื) ืฆื ืึทืืข ืกืขืจืืืขืจืก ืฆื ืื /etc/hosts ืืขืงืข:
172.26.10.106 vhost1
172.26.10.106 vhost2
172.26.10.106 vhost3
172.26.10.106 vhost4ืืื ืืืื ืึทืืฅ ืืื ืืจืืื
nginx -t
systemctl restart nginxืืืฆื ืืึธืื ืก ืื ืกืืึทืืืจื ืขืก ืืื
yum install -y https://packages.timber.io/vector/0.9.X/vector-x86_64.rpmืืึธืืืจ ืืึทืื ืึท ืกืขืืืื ืืก ืืขืงืข ืคึฟืึทืจ systemd /etc/systemd/system/vector.service
[Unit]
Description=Vector
After=network-online.target
Requires=network-online.target
[Service]
User=vector
Group=vector
ExecStart=/usr/bin/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vector
[Install]
WantedBy=multi-user.targetืืื ืงืึทื ืคืืืืขืจ ืื Filebeat ืคืึทืจืืืึทื ืืื ืื /etc/vector/vector.toml config. ืื IP ืึทืืจืขืก 172.26.10.108 ืืื ืื IP ืึทืืจืขืก ืคืื ืื ืงืืึธืฅ ืกืขืจืืืขืจ (ืืืขืงืืึธืจ-ืกืขืจืืืืจืขืจ)
data_dir = "/var/lib/vector"
[sources.nginx_file]
type = "file"
include = [ "/var/log/nginx/access.json.log" ]
start_at_beginning = false
fingerprinting.strategy = "device_and_inode"
[sinks.nginx_output_vector]
type = "vector"
inputs = [ "nginx_file" ]
address = "172.26.10.108:9876"ืื ืืืืกื ื ืืฉื ืคืึทืจืืขืกื ืฆื ืืืืื ืื ืืืขืงืืึธืจ ืืึทื ืืฆืขืจ ืฆื ืื ืคืืจืืื ืื ืืจืืคึผืข ืึทืืื ืึทื ืขืจ ืงืขื ืขื ืืืืขื ืขื ืืึธื ืืขืงืขืก. ืคึฟืึทืจ ืืืึทืฉืคึผืื, nginx ืืื centos ืงืจืืืืฅ ืืึธืืก ืืื ืึทืื ืืจืืคึผืข ืจืขืื.
usermod -a -G adm vectorืืื ืก ืึธื ืืืืื ืื ืืืขืงืืึธืจ ืืื ืกื
systemctl enable vector
systemctl start vectorืืืขืงืืึธืจ ืืึธืืก ืงืขื ืขื ืืืื ืืืืื ืืื ืืึธืก:
journalctl -f -u vectorืขืก ืืึธื ืืืื ืึท ืคึผืึธืืืฆืืข ืืื ืืึธืก ืืื ืื ืืึธืืก
INFO vector::topology::builder: Healthcheck: Passed.ืืจืืง ืืขืกืืื ื
ืืืจ ืืืจืืคืืจื ืืขืกืืื ื ืืื ืึทืคึผืึทืืฉื ืืขื ืืฉืืึทืจืง.
ืื ืืืืคึผื-ืืืฉืืจืื ืคึผืขืงื ืืื ืืื ืกืืึทืืืจื ืืืืฃ ืึทืืข ืกืขืจืืืขืจืก
ืืืจ ืึธื ืืืืื ืืขืกืืื ื ืืื ืึทืคึผืึทืืฉื ืืขื ืืฉืืึทืจืง ืคืื 4 ืคืึทืจืฉืืืขื ืข ืกืขืจืืืขืจืก ืืื ืคืึทืจืฉืืขืื. ืขืจืฉืืขืจ, ืืืจ ืงืึทืืขืจ ืื ืคืึทืจืฉืืขืื ืืืึธืงืืึทื ืืืืืืคึผืืขืงืกืขืจ, ืืื ืืขืืึธืื ืืืจ ืึธื ืืืืื ืืขืกืืื ื ืืื ืื ืึทืคึผืึทืืฉื ืืขื ืืฉืืึทืจืง. ืืื ืฆื ืึทืจืืขืื ืืื ืื ืคืึทืจืฉืืขืื ืืืจ ืงืขื ืขื ืืขืคึฟืื ืขื ืืื .
ืคึฟืื 1 ืกืขืจืืืขืจ
while true; do ab -H "User-Agent: 1server" -c 100 -n 10 -t 10 http://vhost1/; sleep 1; doneืคึฟืื 2 ืกืขืจืืืขืจ
while true; do ab -H "User-Agent: 2server" -c 100 -n 10 -t 10 http://vhost2/; sleep 1; doneืคึฟืื 3 ืกืขืจืืืขืจ
while true; do ab -H "User-Agent: 3server" -c 100 -n 10 -t 10 http://vhost3/; sleep 1; doneืคึฟืื 4 ืกืขืจืืืขืจ
while true; do ab -H "User-Agent: 4server" -c 100 -n 10 -t 10 http://vhost4/; sleep 1; doneืืื ืก ืงืึธื ืืจืึธืืืจื ืื ืืึทืื ืืื ืงืืืงืืึธืืกืข
ืืืื ืฆื Clickhouse
clickhouse-client -h 172.26.10.109 -mืืึทืื ืึท SQL ืึธื ืคึฟืจืขื
SELECT * FROM vector.logs;
โโnode_nameโโโโโฌโโโโโโโโโโโtimestampโโฌโserver_nameโโฌโuser_idโโฌโrequest_fullโโโโฌโrequest_user_agentโโฌโrequest_http_hostโโฌโrequest_uriโโฌโrequest_schemeโโฌโrequest_methodโโฌโrequest_lengthโโฌโrequest_timeโโฌโrequest_referrerโโฌโresponse_statusโโฌโresponse_body_bytes_sentโโฌโresponse_content_typeโโฌโโโremote_addrโโฌโremote_portโโฌโremote_userโโฌโupstream_addrโโฌโupstream_portโโฌโupstream_bytes_receivedโโฌโupstream_bytes_sentโโฌโupstream_cache_statusโโฌโupstream_connect_timeโโฌโupstream_header_timeโโฌโupstream_response_lengthโโฌโupstream_response_timeโโฌโupstream_statusโโฌโupstream_content_typeโโ
โ nginx-vector โ 2020-08-07 04:32:42 โ vhost1 โ โ GET / HTTP/1.0 โ 1server โ vhost1 โ / โ http โ GET โ 66 โ 0.028 โ โ 404 โ 27 โ โ 172.26.10.106 โ 45886 โ โ 172.26.10.106 โ 0 โ 109 โ 97 โ DISABLED โ 0 โ 0.025 โ 27 โ 0.029 โ 404 โ โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโืืขืคึฟืื ืขื ืื ืืจืืืก ืคืื ืืืฉื ืืื Clickhouse
select concat(database, '.', table) as table,
formatReadableSize(sum(bytes)) as size,
sum(rows) as rows,
max(modification_time) as latest_modification,
sum(bytes) as bytes_size,
any(engine) as engine,
formatReadableSize(sum(primary_key_bytes_in_memory)) as primary_keys_size
from system.parts
where active
group by database, table
order by bytes_size desc;ืืึธืืืจ ืืขืคึฟืื ืขื ืืื ืคืื ืืึธืืก ืืขื ืืืขื ืืื Clickhouse.
ืื ืืึธืืก ืืืฉ ืืจืืืก ืืื 857.19 ืืขืืืืืืื.
ืื ืืจืืืก ืคืื ืืขืจ ืืขืืืืงืขืจ ืืึทืื ืืื ืื ืืื ืืขืงืก ืืื Elasticsearch ืืื 4,5GB.
ืืืื ืืืจ ืืึธื ื ืื ืกืคึผืขืฆืืคืืฆืืจื ืืึทืื ืืื ืื ืืืขืงืืึธืจ ืืื ืื ืคึผืึทืจืึทืืขืืขืจืก, ืงืืืงืืึธืืกืข ื ืขืื 4500/857.19 = 5.24 ืืื ืืืืื ืืงืขืจ ืืื ืืื Elasticsearch.
ืืื ืืืขืงืืึธืจ, ืื ืงืึทืืคึผืจืขืฉืึทื ืคืขืื ืืื ืืขื ืืฆื ืืืจื ืคืขืืืงืืึทื.
ืืขืืขืืจืึทื ืฉืืืขืกื ืืืจื
ืืขืืขืืจืึทื ืฉืืืขืกื ืืืจื
ืืขืืขืืจืึทื ืฉืืืขืกื ืืืจื ""
ืืงืืจ: www.habr.com