ืืืคึผืืงืึทืืื, ืืขืฉืขืคื ืคึผืจืึธืืืงืื ืึธืืขืจ ืคืึทืจืืืง ืขืคึฟืขื ืขื-ืืงืืจ ืึทืืืขืจื ืึทืืืืืขืก, ืึทืืึท ืืื Prometheus + Grafana, ืืขื ืขื ืืขื ืืฆื ืฆื ืืึธื ืืืึธืจ ืืื ืึทื ืึทืืืื ืื ืึธืคึผืขืจืึทืฆืืข ืคืื โโNginx. ืืึธืก ืืื ืึท ืืื ืึธืคึผืฆืืข ืคึฟืึทืจ ืืึธื ืืืึธืจืื ื ืึธืืขืจ ืคืึทืงืืืฉ-ืฆืืื ืึทื ืึทืืืืืงืก, ืึธืืขืจ ื ืืฉื ืืืืขืจ ืืึทืงืืืขื ืคึฟืึทืจ ืืืกืืืจืืฉืข ืึทื ืึทืืืกืืก. ืืืืฃ ืงืืื ืคืึธืืงืก ืืืื, ืื ืืึทื ื ืคืื ืืึทืื ืคืื nginx ืืึธืืก ืืื ืืจืึธืืื ื ืจืึทืคึผืึทืืื, ืืื ืฆื ืึทื ืึทืืืื ืึท ืืจืืืก ืกืืืข ืคืื โโืืึทืื, ืขืก ืืื ืืึทืืืฉืืงืึทื ืฆื ื ืืฆื ืขืคึผืขืก ืืขืจ ืกืคึผืขืฉืึทืืืืื.
ืืื ืืขื ืึทืจืืืงื ืืื ืืืขื ืืึธืื ืืืจ ืืื ืืืจ ืงืขื ืขื ื ืืฆื ืฆื ืคืื ืึทื ืืขืจืงืืืึทืื ืืึธืืก, ื ืขืืขื Nginx ืืื ืึท ืืืืฉืคึผืื, ืืื ืืื ืืืขื ืืืืึทืื ืืื ืฆื ืึทืกืขืืืึทื ืึทื ืึทื ืึทืืืกืืก ืืึทืฉืืึธืจื ืคึฟืื ืื ืืึทืื ื ืืฆื ืื ืึธืคึฟื ืืงืืจ cube.js ืคืจืืืืืืขืจืง. ืืึธ ืืื ืื ืคืืืฉืืขื ืืืง ืืืืืื ื ืึทืจืงืึทืืขืงืืฉืขืจ:
TL:DR;
.
ืฆื ืืึทืืืขื ืืื ืคึฟืึธืจืืึทืฆืืข ืืืจ ื ืืฆื , ืคึฟืึทืจ ืคึผืจืึทืกืขืกืื ื - ะธ , ืคึฟืึทืจ ืกืืึธืจืืืืฉ - . ืืื ืืขื ืคึผืขืงื, ืืืจ ืงืขื ืขื ืงืจืึธื ื ืื ืืืืื nginx ืืึธืืก, ืึธืืขืจ ืืืื ืื ืืขืจืข ืืขืฉืขืขื ืืฉื, ืืื ืืขืืื ื ืืื ืืึธืืก ืคืื ืื ืืขืจืข ืืึทืืื ืื ืืก. ืืืจ ืงืขื ืขื ืคืึทืจืืืึทืื ืขืืืขืืข ืคึผืึทืจืฅ ืืื ืขื ืืขื ืคึผืึทืจืฅ ืคึฟืึทืจ ืืืื ืึธื ืืืืื, ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืืจ ืงืขื ืขื ืฉืจืืึทืื ืืึธืืก ืฆื ืงืื ืขืกืืก ืืืืึทื ืคึฟืื nginx, ืืืืคึผืึทืกืื ื ืคืืืขื ืื ืึธืืขืจ ื ืืฆื ืืึธืืกืืึทืฉ ืคึฟืึทืจ ืืขื.
ืงืึทืืขืงืืื ื Nginx ืืึธืืก
ืืืจื ืคืขืืืงืืึทื, Nginx ืืึธืืก ืงืืงื ืขืคึผืขืก ืืื ืืึธืก:
4/9/2019 12:58:17 PM1.1.1.1 - - [09/Apr/2019:09:58:17 +0000] "GET /sign-up HTTP/2.0" 200 9168 "https://example.com/sign-in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
4/9/2019 12:58:17 PM1.1.1.1 - - [09/Apr/2019:09:58:17 +0000] "GET /sign-in HTTP/2.0" 200 9168 "https://example.com/sign-up" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"ืืื ืงืขื ืขื ืืืื ืคึผืึทืจืกืขื, ืึธืืขืจ ืขืก ืืื ืคืื ืืจืื ืืขืจ ืฆื ืคืึทืจืจืืืื ืื Nginx ืงืึทื ืคืืืืขืจืืืฉืึทื ืึทืืื ืึทื ืขืก ืืจืืื ืืึธืืก ืืื JSON:
log_format json_combined escape=json '{ "created_at": "$msec", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"request": "$request", '
'"status": $status, '
'"bytes_sent": $bytes_sent, '
'"request_length": $request_length, '
'"request_time": $request_time, '
'"http_referrer": "$http_referer", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent" }';
access_log /var/log/nginx/access.log json_combined;S3 ืคึฟืึทืจ ืกืืึธืจืืืืฉ
ืฆื ืงืจืึธื ืืึธืืก, ืืืจ ืืืขืื ื ืืฆื S3. ืืึธืก ืึทืืึทืื ืืืจ ืฆื ืงืจืึธื ืืื ืึทื ืึทืืืื ืืึธืืก ืืืืฃ ืืืื ืึธืจื, ืืืืึทื ืึทืืืขื ืึท ืงืขื ืขื ืึทืจืืขืื ืืื ืืึทืื ืืื S3 ืืืืึทื. ืฉืคึผืขืืขืจ ืืื ืืขื ืึทืจืืืงื ืืื ืืืขื ืืึธืื ืืืจ ืืื ืฆื ืืืืื ืืื ืคึผืจืึธืฆืขืก ืืึธืืก ืจืืืืืง, ืึธืืขืจ ืขืจืฉืืขืจ ืืืจ ืืึทืจืคึฟื ืึท ืจืืื ืขืืขืจ ืืื S3, ืืื ืืืึธืก ืืึธืจื ืืฉื ืึทื ืืขืจืฉ ืืืขื ืืืื ืกืืึธืจื. ืขืก ืืื ืืืขืจื ืฆื ืืึทืืจืึทืืื ืืื ืฉืืืึทืื ืืื ืืืึธืก ืืขืื ื ืืืจ ืืืขื ืืึทืื ืืืื ืขืืขืจ, ืืืืึทื ืึทืืืขื ืึท ืืื ื ืืฉื ืื ืืืฆื ืืื ืึทืืข ืืขืื ืื.
ืฉืึทืคึฟื ืึท ืงืจืืึทื ืืื ืื ืึทืืืขื ืึท ืงืึทื ืกืึธืื
ืืึธืืืจ ืืึทืื ืึท ืืืฉ ืืื ืึทืืืขื ืึท ืคึฟืึทืจ ืืึธืืก. ืขืก ืืื ืืืจืฃ ืคึฟืึทืจ ืืืืืข ืฉืจืืืื ืืื ืืืืขื ืขื ืืืื ืืืจ ืคึผืืึทื ืฆื ื ืืฆื Kinesis Firehose. ืขืคึฟืขื ืขื ืื ืึทืืืขื ืึท ืงืึทื ืกืึธืื ืืื ืฉืึทืคึฟื ืึท ืืืฉ:
ืกืงื ืืืฉ ืฉืึทืคืื ื
CREATE EXTERNAL TABLE `kinesis_logs_nginx`(
`created_at` double,
`remote_addr` string,
`remote_user` string,
`request` string,
`status` int,
`bytes_sent` int,
`request_length` int,
`request_time` double,
`http_referrer` string,
`http_x_forwarded_for` string,
`http_user_agent` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
's3://<YOUR-S3-BUCKET>'
TBLPROPERTIES ('has_encrypted_data'='false');ืงืจืืืืืื ื Kinesis Firehose Stream
Kinesis Firehose ืืืขื ืฉืจืืึทืื ืื ืืึทืื ืืืงืืืขื ืคืื Nginx ืฆื S3 ืืื ืื ืืืืกืืขืงืืืื ืคึฟืึธืจืืึทื, ืืืืืืืืื ื ืขืก ืืื ืืืืจืขืงืืขืจืื ืืื ืื YYYY/MM/DD/HH ืคึฟืึธืจืืึทื. ืืึธืก ืืืขื ืืืื ื ืืฆืืง ืืืขื ืืืืขื ืขื ืืึทืื. ืืืจ ืงืขื ื, ืคืื ืงืืจืก, ืฉืจืืึทืื ืืืืึทื ืฆื S3 ืคึฟืื ืคืืืขื ืื, ืึธืืขืจ ืืื ืืขื ืคืึทื ืืืจ ืืืขื ืืึธืื ืฆื ืฉืจืืึทืื JSON, ืืื ืืึธืก ืืื ืืึทืืืึธื ืืฉ ืืืืึทื ืคืื ืื ืืจืืืก ืืจืืืก ืคืื ืื ืืขืงืขืก. ืึทืืืืืืึธื ืึทืืื, ืืืขื ื ืืฆื PrestoDB ืึธืืขืจ Athena, JSON ืืื ืื ืกืืึธืืึทืกื ืืึทืื ืคึฟืึธืจืืึทื. ืึทืืื ืขืคึฟืขื ืขื ืื Kinesis Firehose ืงืึทื ืกืึธืื, ืืื "ืฉืึทืคึฟื ืขืงืกืคึผืจืขืก ืืืึทื", ืกืขืืขืงืืืจื "ืืืจืขืงื PUT" ืืื ืื "ืขืงืกืคึผืจืขืก" ืคืขืื:
ืืื ืืขืจ ืืืืึทืืขืจ ืงืืืืื, ืกืขืืขืงืืืจื "ืจืขืงืึธืจื ืคึฟืึธืจืืึทื ืงืึทื ืืืขืจืืฉืึทื" - "ืขื ืึทืืืขื" ืืื ืกืขืืขืงืืืจื "ืึทืคึผืึทืืฉื ORC" ืืื ืื ืจืขืงืึธืจืืื ื ืคึฟืึธืจืืึทื. ืืืื ืฆื ืขืืืขืืข ืคืึธืจืฉืื ื , ืืึธืก ืืื ืื ืึธืคึผืืืืึทื ืคึฟืึธืจืืึทื ืคึฟืึทืจ PrestoDB ืืื Athena. ืืืจ ื ืืฆื ืื ืืืฉ ืืืึธืก ืืืจ ืืืฉืืคื ืืืืื ืืื ืึท ืกืืฉืขืืึท. ืืืืข ืืึธื ืึทื ืืืจ ืงืขื ืขื ืกืคึผืขืฆืืคืืฆืืจื ืงืืื S3 ืึธืจื ืืื ืงืื ืขืกืืก; ืืืืื ืื ืกืืฉืขืืึท ืืื ืืขื ืืฆื ืคึฟืื ืื ืืืฉ. ืึธืืขืจ ืืืื ืืืจ ืกืคึผืขืฆืืคืืฆืืจื ืึท ืึทื ืืขืจืฉ S3 ืึธืจื, ืืืจ ืืืขื ื ืืฉื ืงืขื ืขื ืฆื ืืืืขื ืขื ืื ืจืขืงืึธืจืืก ืคึฟืื ืืขื ืืืฉ.
ืืืจ ืกืขืืขืงืืืจื S3 ืคึฟืึทืจ ืกืืึธืจืืืืฉ ืืื ืื ืขืืขืจ ืืืึธืก ืืืจ ืืืฉืืคื ืคืจืืขืจ. Aws Glue Crawler, ืืืึธืก ืืื ืืืขื ืจืขืื ืืืขืื ืึท ืืืกื ืฉืคึผืขืืขืจ, ืงืขื ื ืืฉื ืึทืจืืขืื ืืื ืคึผืจืขืคืืงืก ืืื ืึท S3 ืขืืขืจ, ืึทืืื ืขืก ืืื ืืืืืืืง ืฆื ืืึธืื ืขืก ืืืืืืง.
ืื ืจืืขื ืึธืคึผืฆืืขืก ืงืขื ืขื ืืืื ืคืืจืขื ืืขืจื ืืืคึผืขื ืืื ื ืืืืฃ ืืืื ืืึทืกืข; ืืื ืืืืฉืึทืืืึทืื ื ืืฆื ืื ืคืขืืืงืืึทื ืึธื ืขืก. ืืึทืืขืจืงืื ื ืึทื S3 ืงืึทืืคึผืจืขืฉืึทื ืืื ื ืื ืื ืืืฆื, ืึธืืขืจ ORC ื ืืฆื ืืขืืืืจื ืงืึทืืคึผืจืขืฉืึทื ืืืจื ืคืขืืืงืืึทื.
ืคืืืขื ืื
ืืืฆื ืึทื ืืืจ ืืึธืื ืงืึทื ืคืืืืขืจื ืกืืึธืจืื ื ืืื ืจืืกืืืืื ื ืืึธืืก, ืืืจ ืืึทืจืคึฟื ืฆื ืงืึทื ืคืืืืขืจ ืฉืืงื. ืืืจ ืืืขืื ื ืืฆื , ืืืืึทื ืืื ืืืืข ืจืืื, ืึธืืขืจ ืืืจ ืงืขื ืขื ื ืืฆื Logstash ืึธืืขืจ ืฉืืงื ืืึธืืก ืฆื ืงืื ืขืกืืก ืืืืึทื. ืื ืคืืืขื ืื ืกืขืจืืืขืจ ืงืขื ืขื ืืืื ืืึธื ืืฉื ืืื ืขืืืขืืข ืืืขืื, ืืื ืืืขื ืืึธืื ืืืจ ืืืขืื ืืึธืงืขืจ ืืืืึทื ืขืก ืืื ืคึผืฉืื ืืื ืืึทืงืืืขื.
ืขืจืฉืืขืจ, ืืืจ ืืึทืจืคึฟื ืื fluent.conf ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข. ืฉืึทืคึฟื ืขืก ืืื ืืืืื ืืงืืจ:
ืคืึธืจืืืก
ืคึผืึธืจื ืงืกื ืืืงืก
ืืื ืื 0.0.0.0
ืืืฆื ืืืจ ืงืขื ืขื ืึธื ืืืืื ืื Fluentd ืกืขืจืืืขืจ. ืืืื ืืืจ ืืึทืจืคึฟื ืึท ืืขืจ ืึทืืืึทื ืกืืจืืข ืงืึทื ืคืืืืขืจืืืฉืึทื, ืืืื ืฆื ืขืก ืืื ืึท ืืืืืืื ืืืขืืืืืึทืืขืจ, ืึทืจืืึทื ืืขืจืขืื ื ืืื ืฆื ืึทืกืขืืืึทื ืืืื ืืืื.
$ docker run
-d
-p 24224:24224
-p 24224:24224/udp
-v /data:/fluentd/log
-v <PATH-TO-FLUENT-CONF>:/fluentd/etc fluentd
-c /fluentd/etc/fluent.conf
fluent/fluentd:stableืืขื ืงืึทื ืคืืืืขืจืืืฉืึทื ื ืืฆื ืืขื ืืจื /fluentd/log ืฆื ืงืึทืฉ ืืึธืืก ืืืืืขืจ ืฉืืงื. ืืืจ ืงืขื ืขื ืืึธื ืึธื ืืขื, ืึธืืขืจ ืืืขื ืืืจ ืจืืกืืึทืจื, ืืืจ ืงืขื ืขื ืคืึทืจืืืจื ืึทืืฅ ืงืึทืฉื ืืื ืฆืืจืืง-ืืจืืืงืื ื ืึทืจืืขื. ืืืจ ืงืขื ื ืืืื ื ืืฆื ืงืืื ืคึผืึธืจื; 24224 ืืื ืื ืคืขืืืงืืึทื ืคืืืขื ืื ืคึผืึธืจื.
ืืืฆื ืึทื ืืืจ ืืืืคื Fluent, ืืืจ ืงืขื ืขื ืฉืืงื Nginx ืืึธืืก ืืึธืจื. ืืืจ ืืืืฉืึทืืืึทืื ืืืืคื Nginx ืืื ืึท ืืึธืงืงืขืจ ืงืึทื ืืืื ืขืจ, ืืื ืืืึธืก ืคืึทื ืืึธืงืขืจ ืืื ืึท ืืขืืืืจื ืืึธืืื ื ืฉืึธืคืขืจ ืคึฟืึทืจ ืคืืืขื ืื:
$ docker run
--log-driver=fluentd
--log-opt fluentd-address=<FLUENTD-SERVER-ADDRESS>
--log-opt tag="{{.Name}}"
-v /some/content:/usr/share/nginx/html:ro
-d
nginxืืืื ืืืจ ืืืืคื Nginx ืึทื ืืขืจืฉ, ืืืจ ืงืขื ืขื ื ืืฆื ืงืืึธืฅ ืืขืงืขืก, Fluentd ืืื .
ืืึธืืืจ ืืืืื ืืขื ืงืืึธืฅ ืคึผืึทืจืกืื ื ืงืึทื ืคืืืืขืจื ืืืืื ืฆื ืื ืคืืืึทื ื ืงืึทื ืคืืืืขืจืืืฉืึทื:
<filter YOUR-NGINX-TAG.*>
@type parser
key_name log
emit_invalid_record_to_error false
<parse>
@type json
</parse>
</filter>ืืื ืฉืืงื ืืึธืืก ืฆื ืงืื ืขืกืืก ื ืืฆื :
<match YOUR-NGINX-TAG.*>
@type kinesis_firehose
region region
delivery_stream_name <YOUR-KINESIS-STREAM-NAME>
aws_key_id <YOUR-AWS-KEY-ID>
aws_sec_key <YOUR_AWS-SEC_KEY>
</match>ืึทืืืขื ืึท
ืืืื ืืืจ ืืึธื ืงืึทื ืคืืืืขืจื ืึทืืฅ ืจืืืืืง, ื ืึธื ืึท ืืฉืขืช (ืืืจื ืคืขืืืงืืึทื, ืงืื ืขืกืืก ืจืขืงืึธืจืืก ืืืงืืืขื ืืึทืื ืึทืืึธื ืืขืืขืจ 10 ืืื ืื) ืืืจ ืืึธื ืืขื ืืึธื ืืขืงืขืก ืืื S3. ืืื ืื "ืืึธื ืืืึธืจืื ื" ืืขื ืื ืคืื Kinesis Firehose ืืืจ ืงืขื ืขื ืืขื ืืื ืคืื ืืึทืื ืืขื ืขื ืจืขืงืึธืจืืขื ืืื S3, ืืื ืืขืืื ื ืืื ืขืจืจืึธืจืก. ืื ืืืืกื ื ืืฉื ืคืึทืจืืขืกื ืฆื ืืขืื ืฉืจืืืื ืึทืงืกืขืก ืฆื ืื S3 ืขืืขืจ ืฆื ืื ืงืื ืขืกืืก ืจืึธืืข. ืืืื ืงืื ืขืกืืก ืงืขื ื ืืฉื ืคึผืึทืจืกืืจื ืขืคึผืขืก, ืขืก ืืืขื ืืืืื ืื ืขืจืจืึธืจืก ืฆื ืืขืจ ืืขืืืืงืขืจ ืขืืขืจ.
ืืืฆื ืืืจ ืงืขื ืขื ืืขื ืื ืืึทืื ืืื ืึทืืืขื ืึท. ืืึธืืืจ ืืขืคึฟืื ืขื ืื ืืขืฆืืข ืจืืงืืืขืก ืคึฟืึทืจ ืืืึธืก ืืืจ ืืึธืื ืืืืืขืงืขืจื ืขืจืจืึธืจืก:
SELECT * FROM "db_name"."table_name" WHERE status > 499 ORDER BY created_at DESC limit 10;ืกืงืึทื ืื ื ืึทืืข ืจืขืงืึธืจืืก ืคึฟืึทืจ ืืขืืขืจ ืืงืฉื
ืืืฆื ืืื ืืืขืจ ืืึธืืก ืืขื ืขื ืคึผืจืึทืกืขืกื ืืื ืกืืึธืจื ืืื S3 ืืื ORC, ืงืึทืืคึผืจืขืกื ืืื ืืจืืื ืคึฟืึทืจ ืึทื ืึทืืืกืืก. Kinesis Firehose ืืคืืื ืึธืจืืึทื ืืืืจื ืืื ืืื ืืืจืขืงืืขืจืื ืคึฟืึทืจ ืืขืืขืจ ืฉืขื. ืึธืืขืจ, ืืื ืืึทื ื ืืื ืื ืืืฉ ืืื ื ืืฉื ืคึผืึทืจืืืฉืึทื ื, ืึทืืืขื ืึท ืืืขื ืืึธืื ืึทืืข-ืฆืืื ืืึทืื ืืืืฃ ืืขืืขืจ ืืงืฉื, ืืื ืืขืืื ืืืืกื ืขืืขื. ืืึธืก ืืื ืึท ืืจืืืก ืคึผืจืึธืืืขื ืคึฟืึทืจ ืฆืืืื ืกืืืืช:
- ืืขืจ ืืึทื ื ืคืื ืืึทืื ืืื ืงืขืกืืืืขืจ ืืจืึธืืื ื, ืกืืึธืืื ื ืึทืจืึธืคึผ ืงืืืืจืื;
- ืึทืืืขื ืึท ืืื ืืืืขื ืืืืืจื ืืืืฃ ืื ืืึทื ื ืคืื ืืึทืื ืกืงืึทื ื, ืืื ืึท ืืื ืืืื ืคืื 10 ืื ืคึผืขืจ ืืงืฉื.
ืฆื ืคืึทืจืจืืืื ืืขื, ืืืจ ื ืืฆื AWS Glue Crawler, ืืืึธืก ืืืขื ืงืจืืื ืื ืืึทืื ืืื S3 ืืื ืฉืจืืึทืื ืื ืฆืขืืืืืื ื ืืื ืคึฟืึธืจืืึทืฆืืข ืฆื ืื Glue Metastore. ืืึธืก ืืืขื ืืึธืื ืืื ืื ืฆื ื ืืฆื ืคึผืึทืจืืืฉืึทื ื ืืื ืึท ืคืืืืขืจ ืืืขื ืืืจ ืคืจืขืื ืึทืืืขื ืึท, ืืื ืขืก ืืืขื ื ืึธืจ ืืืขืจืงืืงื ืื ืืืจืขืงืืขืจืื ืกืคึผืขืกืืคืืขื ืืื ืื ืึธื ืคึฟืจืขื.
ืืึทืฉืืขืืืงื Amazon Glue Crawler
Amazon Glue Crawler ืกืงืึทื ื ืึทืืข ืื ืืึทืื ืืื ืื S3 ืขืืขืจ ืืื ืงืจืืืืฅ ืืืฉื ืืื ืคึผืึทืจืืืฉืึทื ื. ืฉืึทืคึฟื ืึท ืงืืื ืงืจืึทืืืืขืจ ืคึฟืื ืื AWS ืงืืื ืงืึทื ืกืึธืื ืืื ืืืืื ืึท ืขืืขืจ ืืื ืืืจ ืงืจืึธื ืื ืืึทืื. ืืืจ ืงืขื ืขื ื ืืฆื ืืืื ืงืจืึทืืืืขืจ ืคึฟืึทืจ ืขืืืขืืข ืืึทืงืึทืฅ, ืืื ืืืึธืก ืคืึทื ืขืก ืืืขื ืืึทืื ืืืฉื ืืื ืื ืกืคึผืขืกืืคืืขื ืืึทืืึทืืืืก ืืื ื ืขืืขื ืืืึธืก ืืืืึทืื ืื ื ืขืืขื ืคืื ืื ืืึทืงืึทืฅ. ืืืื ืืืจ ืคึผืืึทื ืฆื ื ืืฆื ืื ืืึทืื ืงืขืกืืืืขืจ, ืืืื ืืืืขืจ ืฆื ืงืึทื ืคืืืืขืจ Crawler ืก ืงืึทืืขืจ ืคึผืืึทื ืฆื ืคึผืึทืกื ืืืื ืืึทืจืฃ. ืืืจ ื ืืฆื ืืืื ืงืจืึทืืืืขืจ ืคึฟืึทืจ ืึทืืข ืืืฉื, ืืืึธืก ืืืืคื ืืขืืขืจ ืฉืขื.
ืฆืขืืืืื ืืืฉื
ื ืึธื ืืขืจ ืขืจืฉืืขืจ ืงืึทืืขืจ ืคืื ืื ืงืจืึทืืืืขืจ, ืื ืืืฉื ืคึฟืึทืจ ืืขืืขืจ ืกืงืึทื ื ืขืืขืจ ืืึธื ืืืื ืืขืืืืื ืืื ืื ืืึทืืึทืืืืก ืกืคึผืขืกืืคืืขื ืืื ืื ืกืขืืืื ืืก. ืขืคึฟืขื ืขื ืื ืึทืืืขื ืึท ืงืึทื ืกืึธืื ืืื ืืขืคึฟืื ืขื ืื ืืืฉ ืืื Nginx ืืึธืืก. ืืึธืืืจ ืคึผืจืืืืจื ืขืคึผืขืก ืืืืขื ืขื:
SELECT * FROM "default"."part_demo_kinesis_bucket"
WHERE(
partition_0 = '2019' AND
partition_1 = '04' AND
partition_2 = '08' AND
partition_3 = '06'
);ืื ืึธื ืคึฟืจืขื ืืืขื ืืืืกืงืืืึทืื ืึทืืข ืจืขืงืึธืจืืก ืืืงืืืขื ืฆืืืืฉื 6:7 ืืื 8:2019 ืืืืฃ ืืคืจืื XNUMX, XNUMX. ืืืขืจ ืืื ืคืื ืืขืจ ืขืคืขืงืืืื ืืื ืืึธืก ืืื ื ืึธืจ ืืืืขื ืขื ืคืื ืึท ื ืื-ืคึผืึทืจืืืืฉืึทื ื ืืืฉ? ืืึธืืืจ ืืขืคึฟืื ืขื ืืืืก ืืื ืกืขืืขืงืืืจื ืื ืืขืืืข ืจืขืงืึธืจืืก, ืคืืืืขืจืื ื ืืื ืืืจื ืืืืขืกืืึทืืคึผ:
3.59 ืกืขืงืื ืืขืก ืืื 244.34 ืืขืืืืืื ืคืื ืืึทืื ืืืืฃ ืึท ืืึทืืึทืกืขื ืืื ืืืืื ืึท ืืืึธื ืคืื ืืึธืืก. ืืึธืืืจ ืคึผืจืืืืจื ืึท ืคืืืืขืจ ืืืจื ืฆืขืืืืืื ื:
ื ืืืกื ืคืึทืกืืขืจ, ืึธืืขืจ ืจืืึฟ ืืืคึผืึธืจืืึทื ืืื - ืืืืื 1.23 ืืขืืืืืื ืคืื ืืึทืื! ืขืก ืืืึธืื ืืืื ืคืื ืืฉืืคึผืขืจ ืืืื ื ืืฉื ืคึฟืึทืจ ืื ืืื ืืืื 10 ืืขืืืืืื ืคึผืขืจ ืืงืฉื ืืื ืื ืคึผืจืืืกืื ื. ืึธืืขืจ ืขืก ืืื ื ืึธื ืคืื ืืขืกืขืจ, ืืื ืืื ืืจืืืก ืืึทืืึทืกืขืฅ ืื ืืืืืง ืืืขื ืืืื ืคืื ืืขืจ ืืืคึผืจืขืกืืื.
ืืืืขื ืึท ืืึทืฉืืึธืจื ื ืืฆื Cube.js
ืฆื ืึทืกืขืืืึทื ืื ืืึทืฉืืึธืจื, ืืืจ ื ืืฆื ืื Cube.js ืึทื ืึทืืืกืืก ืคืจืืืืืืขืจืง. ืขืก ืืื ืืึทื ืฅ ืึท ืคึผืืึทืฅ ืคืื ืคืึทื ืืงืฉืึทื ื, ืึธืืขืจ ืืืจ ืืขื ืขื ืืื ืืขืจืขืกืืจื ืืื ืฆืืืื: ืื ืคืืืืงืืื ืฆื ืืืืืึธืืึทืืืฉ ื ืืฆื ืฆืขืืืืืื ื ืคืืืืขืจืก ืืื ืืึทืื ืคืึทืจ - ืึทืืืจืขืืึทืืืึธื. ืขืก ื ืืฆื ืืึทืื ืกืืฉืขืืึท , ืืขืฉืจืืื ืืื ืืืฉืึทืืืึทืกืงืจืืคึผื ืฆื ืืืฉืขื ืขืจืืื ืกืงื ืืื ืืืกืคืืจื ืึท ืืึทืืึทืืืืก ืึธื ืคึฟืจืขื. ืืืจ ื ืึธืจ ืืึทืจืคึฟื ืฆื ืึธื ืืืืึทืื ืืื ืฆื ื ืืฆื ืื ืฆืขืืืืืื ื ืคืืืืขืจ ืืื ืื ืืึทืื ืกืืฉืขืืึท.
ืืึธืืืจ ืฉืึทืคึฟื ืึท ื ืืึทืข Cube.js ืึทืคึผืืึทืงืืืฉืึทื. ืืื ื ืืืจ ืฉืืื ื ืืฆื ืื AWS ืึธื ืืืืื, ืขืก ืืื ืืึทืืืฉืืงืึทื ืฆื ื ืืฆื Lambda ืคึฟืึทืจ ืืืคึผืืืืืึทื ื. ืืืจ ืงืขื ื ื ืืฆื ืื ืขืงืกืคึผืจืขืกืก ืืืกืืขืจ ืคึฟืึทืจ ืืืจ ืืืื ืืืจ ืคึผืืึทื ืฆื ืืึทืืขืืึธืก ืื Cube.js ืืึทืงืขื ื ืืื Heroku ืึธืืขืจ Docker. ืื ืืึทืงืืืืขื ืืืืฉืึทื ืืืฉืจืืืื ืื ืืขืจืข .
$ npm install -g cubejs-cli
$ cubejs create nginx-log-analytics -t serverless -d athenaืกืืืืืืข ืืืขืจืืึทืืึทืื ืืขื ืขื ืืขื ืืฆื ืฆื ืงืึทื ืคืืืืขืจ ืืืืืึทืืืืก ืึทืงืกืขืก ืืื cube.js. ืืขืจ ืืขื ืขืจืึทืืึธืจ ืืืขื ืฉืึทืคึฟื ืึท .ืขื ืื ืืขืงืข ืืื ืืืึธืก ืืืจ ืงืขื ืขื ืกืคึผืขืฆืืคืืฆืืจื ืืืื ืฉืืืกืืขื ืคึฟืึทืจ .
ืืืฆื ืืืจ ืืึทืจืคึฟื , ืืื ืืืึธืก ืืืจ ืืืขืื ืึธื ืืืืึทืื ืคึผืื ืงื ืืื ืืื ืืืขืจ ืืึธืืก ืืขื ืขื ืกืืึธืจื. ืขืก ืืืจ ืงืขื ืขื ืืืื ืกืคึผืขืฆืืคืืฆืืจื ืืื ืฆื ืจืขืืขื ืขื ืืขืืจืืงืก ืคึฟืึทืจ ืืึทืฉืืึธืจืื.
ืืื ืืขืจ ืืืขืืืืืึทืืขืจ schema, ืฉืึทืคึฟื ืึท ืืขืงืข Logs.js. ืืึธ ืืื ืึท ืืืืฉืคึผืื ืืึทืื ืืึธืืขื ืคึฟืึทืจ nginx:
ืืึธืืขื ืงืึธื
const partitionFilter = (from, to) => `
date(from_iso8601_timestamp(${from})) <= date_parse(partition_0 || partition_1 || partition_2, '%Y%m%d') AND
date(from_iso8601_timestamp(${to})) >= date_parse(partition_0 || partition_1 || partition_2, '%Y%m%d')
`
cube(`Logs`, {
sql: `
select * from part_demo_kinesis_bucket
WHERE ${FILTER_PARAMS.Logs.createdAt.filter(partitionFilter)}
`,
measures: {
count: {
type: `count`,
},
errorCount: {
type: `count`,
filters: [
{ sql: `${CUBE.isError} = 'Yes'` }
]
},
errorRate: {
type: `number`,
sql: `100.0 * ${errorCount} / ${count}`,
format: `percent`
}
},
dimensions: {
status: {
sql: `status`,
type: `number`
},
isError: {
type: `string`,
case: {
when: [{
sql: `${CUBE}.status >= 400`, label: `Yes`
}],
else: { label: `No` }
}
},
createdAt: {
sql: `from_unixtime(created_at)`,
type: `time`
}
}
});ืืึธ ืืืจ ืืขื ืขื ื ืืฆื ืื ืืืึทืืขืืืืืง ืฆื ืืืฉืขื ืขืจืืื ืึท SQL ืึธื ืคึฟืจืขื ืืื ืึท ืฆืขืืืืืื ื ืคืืืืขืจ.
ืืืจ ืืืื ืฉืืขืื ืื ืืขืืจืืงืก ืืื ืคึผืึทืจืึทืืขืืขืจืก ืืืึธืก ืืืจ ืืืืื ืฆื ืืืืึทืื ืืืืฃ ืื ืืึทืฉืืึธืจื ืืื ืกืคึผืขืฆืืคืืฆืืจื ืคืึทืจ-ืึทืืืืจืขืืึทืืืึธื ืก. Cube.js ืืืขื ืฉืึทืคึฟื ื ืึธื ืืืฉื ืืื ืคืึทืจ-ืึทืืืึทืืืฉืืืืึทื ืืึทืื ืืื ืืืขื ืืืืืึธืืึทืืืฉ ืืขืจืืืึทื ืืืงื ืื ืืึทืื ืืืขื ืขืก ืงืืื. ืืึธืก ื ืื ืืืืื ืกืคึผืืื ืึทืจืืืฃ ืคึฟืจืืื, ืึธืืขืจ ืืืื ืจืึทืืืกืึทื ืื ืงืึธืก ืคืื ื ืืฆื ืึทืืืขื ืึท.
ืืึธืืืจ ืืืืื ืื ืืื ืคึฟืึธืจืืึทืฆืืข ืฆื ืื ืืึทืื ืกืืฉืขืืึท ืืขืงืข:
preAggregations: {
main: {
type: `rollup`,
measureReferences: [count, errorCount],
dimensionReferences: [isError, status],
timeDimensionReference: createdAt,
granularity: `day`,
partitionGranularity: `month`,
refreshKey: {
sql: FILTER_PARAMS.Logs.createdAt.filter((from, to) =>
`select
CASE WHEN from_iso8601_timestamp(${to}) + interval '3' day > now()
THEN date_trunc('hour', now()) END`
)
}
}
}ืืืจ ืกืคึผืขืฆืืคืืฆืืจื ืืื ืืขื ืืึธืืขื ืึทื ืขืก ืืื ื ืืืืืง ืฆื ืคืึทืจ - ืึทืืืจืขืืึทืืขื ืืึทืื ืคึฟืึทืจ ืึทืืข ืืขืืจืืงืก ืืขื ืืฆื, ืืื ื ืืฆื ืคึผืึทืจืืืฉืึทื ืื ื ืืืจื ืืืืฉ. ืงืขื ืขื ืืืืืืืืง ืคืึทืจืืืืขืจื ืืึทืื ืืึทืืืื ื ืืื ืึทืคึผืืืืืื ื.
ืืืฆื ืืืจ ืงืขื ืขื ืึทืกืขืืืึทื ืื ืืึทืฉืืึธืจื!
Cube.js ืืึทืงืขื ื ืืื ืืื ืึท ืืึทื ื ืคืื ืงืืืขื ื ืืืืืจืขืจืื ืคึฟืึทืจ ืคืึธืืงืก ืคืจืึธื ื-ืกืืฃ ืคืจืึทืืขืืืึธืจืงืก. ืืืจ ืืืขืื ื ืืฆื ืื ืจืขืึทืงื ืืืขืจืกืืข ืคืื โโโโืืขื ืงืืืขื ื ืฆื ืืืืขื ืื ืืึทืฉืืึธืจื. Cube.js ืืื ืืืืื ืืึทืื, ืึทืืื ืืืจ ืืึทืจืคึฟื ืึท ืืืืืฉืืืึทืืึทืืืืฉืึทื ืืืืืืึธืืขืง - ืืื ืืื ืขืก , ืึธืืขืจ ืืืจ ืงืขื ืขื ื ืืฆื ืงืืื.
ืื Cube.js ืกืขืจืืืขืจ ืึทืงืกืขืคึผืฅ ืื ืืงืฉื ืืื , ืืืึธืก ืกืคึผืขืฆืืคืืฆืืจื ืื ืคืืจืืื ืื ืืขืืจืืงืก. ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืฆื ืจืขืืขื ืขื ืืื ืคืืืข ืขืจืจืึธืจืก Nginx ืืขืืขืื ืืืจื ืืึธื, ืืืจ ืืึทืจืคึฟื ืฆื ืฉืืงื ืื ืคืืืืขื ืืข ืืขืื:
{
"measures": ["Logs.errorCount"],
"timeDimensions": [
{
"dimension": "Logs.createdAt",
"dateRange": ["2019-01-01", "2019-01-07"],
"granularity": "day"
}
]
}ืืึธืืืจ ืื ืกืืึทืืืจื ืื Cube.js ืงืืืขื ื ืืื ืื ืจืขืึทืงื ืงืึธืืคึผืึธื ืขื ื ืืืืืืึธืืขืง ืืืจื NPM:
$ npm i --save @cubejs-client/core @cubejs-client/reactืืืจ ืึทืจืืึทื ืคืืจ ืงืึทืืคึผืึธืื ืึทื ืฅ cubejs ะธ QueryRendererืฆื ืึธืคึผืืึธืืืจื ืื ืืึทืื ืืื ืงืืืึทืื ืื ืืึทืฉืืึธืจื:
ืืึทืฉืืึธืจื ืงืึธื
import React from 'react';
import { LineChart, Line, XAxis, YAxis } from 'recharts';
import cubejs from '@cubejs-client/core';
import { QueryRenderer } from '@cubejs-client/react';
const cubejsApi = cubejs(
'YOUR-CUBEJS-API-TOKEN',
{ apiUrl: 'http://localhost:4000/cubejs-api/v1' },
);
export default () => {
return (
<QueryRenderer
query={{
measures: ['Logs.errorCount'],
timeDimensions: [{
dimension: 'Logs.createdAt',
dateRange: ['2019-01-01', '2019-01-07'],
granularity: 'day'
}]
}}
cubejsApi={cubejsApi}
render={({ resultSet }) => {
if (!resultSet) {
return 'Loading...';
}
return (
<LineChart data={resultSet.rawData()}>
<XAxis dataKey="Logs.createdAt"/>
<YAxis/>
<Line type="monotone" dataKey="Logs.errorCount" stroke="#8884d8"/>
</LineChart>
);
}}
/>
)
}ืืึทืฉืืึธืจื ืงืืืืื ืืขื ืขื ืื ืืืฆื ืืื .
ืืงืืจ: www.habr.com