ืคืœื•ืขื ื˜ื“: ืคืืจื•ื•ืืก ืขืก ืื™ื– ื•ื•ื™ื›ื˜ื™ืง ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจ

ืคืœื•ืขื ื˜ื“: ืคืืจื•ื•ืืก ืขืก ืื™ื– ื•ื•ื™ื›ื˜ื™ืง ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจ

ื ืึธื•ื•ืึทื“ื™ื™ึทืก, ืขืก ืื™ื– ืื•ืžืžืขื’ืœืขืš ืฆื• ื™ืžืึทื“ื–ืฉืึทืŸ ืึท ืงื•ื‘ืขืจื ืขื˜ืขืก-ื‘ืื–ื™ืจื˜ ืคึผืจื•ื™ืขืงื˜ ืึธืŸ ื“ื™ ELK ืึธื ืœื™ื™ื’ืŸ, ื•ื•ืึธืก ืกืึทื•ื•ืขืก ืœืึธื’ืก ืคื•ืŸ ื‘ื™ื™ื“ืข ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ืื•ืŸ ืกื™ืกื˜ืขื ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืคื•ืŸ ื“ืขื ืงื ื•ื™ืœ. ืื™ืŸ ืื•ื ื“ื–ืขืจ ืคื™ืจ, ืžื™ืจ ื ื•ืฆืŸ ื“ื™ EFK ืึธื ืœื™ื™ื’ืŸ ืžื™ื˜ Fluentd ืึทื ืฉื˜ืึธื˜ ืคื•ืŸ Logstash.

Fluentd ืื™ื– ืึท ืžืึธื“ืขืจืŸ, ื•ื ื™ื•ื•ืขืจืกืึทืœ ืงืœืึธืฅ ืงืึทืœืขืงื˜ืขืจ ื•ื•ืึธืก ืื™ื– ื’ื™ื™ื ื™ื ื’ ืžืขืจ ืื•ืŸ ืžืขืจ ืคึผืึธืคึผื•ืœืึทืจื™ื˜ืขื˜ ืื•ืŸ ื–ื™ืš ืื™ื™ื ื’ืขืฉืจื™ื‘ืŸ ื“ื™ ืงืœืึธื•ื“ ื ืึทื˜ื™ื•ื•ืข ืงืึทืžืคึผื™ื•ื˜ื™ื ื’ ื•ื•ื™ืงื™ืคึผืขื“ื™ืข, ื•ื•ืึธืก ืื™ื– ื•ื•ืึธืก ื–ื™ื™ืŸ ืึทื ื˜ื•ื•ื™ืงืœื•ื ื’ ื•ื•ืขืงื˜ืึธืจ ืื™ื– ืคืึธื•ืงื™ืกื˜ ืื•ื™ืฃ ื ื•ืฆืŸ ืื™ืŸ ืงืึทื ื“ื–ืฉืึทื ื’ืงืฉืึทืŸ ืžื™ื˜ Kubernetes.

ื“ืขืจ ืคืึทืงื˜ ืคื•ืŸ ื ื™ืฆืŸ Fluentd ืึทื ืฉื˜ืึธื˜ ืคื•ืŸ Logstash ื˜ื•ื˜ ื ื™ืฉื˜ ื˜ื•ื™ืฉืŸ ื“ื™ ืึทืœื’ืขืžื™ื™ื ืข ืขืกืึทื ืก ืคื•ืŸ ื“ื™ ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืคึผืขืงืœ, ืึธื‘ืขืจ, Fluentd ืื™ื– ืงืขืจืึทืงื˜ืขืจื™ื™ื–ื“ ื“ื•ืจืš ื–ื™ื™ืŸ ืื™ื™ื’ืขื ืข ืกืคึผืขืฆื™ืคื™ืฉ ื ื•ืึทื ืกื™ื– ืจื™ื–ืึทืœื˜ื™ื ื’ ืคื•ืŸ ื–ื™ื™ืŸ ื•ื•ืขืจืกืึทื˜ื™ืœืึทื˜ื™.

ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ื•ื•ืขืŸ ืžื™ืจ ืกื˜ืึทืจื˜ืขื“ ื ื™ืฆืŸ EFK ืื™ืŸ ืึท ืคืึทืจื ื•ืžืขืŸ ืคึผืจื•ื™ืขืงื˜ ืžื™ื˜ ืึท ื”ื•ื™ืš ื™ื ื˜ืขื ืกื™ื˜ื™ ืคื•ืŸ ืœืึธื’ื™ื ื’, ืžื™ืจ ื–ืขื ืขืŸ ืคื™ื™ืกื˜ ืžื™ื˜ ื“ื™ ืคืึทืงื˜ ืึทื– ืื™ืŸ ืงื™ื‘ืึทื ืึท ืขื˜ืœืขื›ืข ืึทืจื˜ื™ืงืœืขืŸ ื–ืขื ืขืŸ ื’ืขื•ื•ื™ื–ืŸ ืขื˜ืœืขื›ืข ืžืึธืœ ืขื˜ืœืขื›ืข ืžืึธืœ. ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ ืžื™ืจ ื•ื•ืขื˜ ื–ืึธื’ืŸ ืื™ืจ ื•ื•ืึธืก ื“ืขื ื“ืขืจืฉื™ื™ื ื•ื ื’ ืึทืงืขืจื– ืื•ืŸ ื•ื•ื™ ืฆื• ืกืึธืœื•ื•ืข ื“ื™ ืคึผืจืึธื‘ืœืขื.

ื“ื™ ืคึผืจืึธื‘ืœืขื ืคื•ืŸ ื“ืึธืงื•ืžืขื ื˜ ื“ื•ืคึผืœื™ืงืึทื˜ื™ืึธืŸ

ืื™ืŸ ืื•ื ื“ื–ืขืจ ืคึผืจืึทื“ื–ืฉืขืงืก, Fluentd ืื™ื– ื“ื™ืคึผืœื•ื™ื“ ื•ื•ื™ ืึท DaemonSet (ืึธื˜ืึทืžืึทื˜ื™ืง ืœืึธื ื˜ืฉื˜ ืื™ืŸ ืื™ื™ืŸ ื‘ื™ื™ึทืฉืคึผื™ืœ ืื•ื™ืฃ ื™ืขื“ืขืจ ื ืึธื“ืข ืคื•ืŸ โ€‹โ€‹ื“ื™ Kubernetes ืงื ื•ื™ืœ) ืื•ืŸ ืžืึธื ื™ื˜ืึธืจ ืกื˜ื“ืึธื•ื˜ ืงืึทื ื˜ื™ื™ื ืขืจ ืœืึธื’ืก ืื™ืŸ /var/log/containers. ื ืึธืš ื–ืึทืžืœื•ื ื’ ืื•ืŸ ืคึผืจืึทืกืขืกื™ื ื’, ื“ื™ ืœืึธื’ืก ืื™ืŸ ื“ื™ ืคืึธืจืขื ืคื•ืŸ JSON ื“ืึธืงื•ืžืขื ื˜ืŸ ื–ืขื ืขืŸ ื’ืขืฉื™ืงื˜ ืฆื• ElasticSearch, ืื•ื™ืคืฉื˜ื™ื™ืŸ ืื™ืŸ ืงื ื•ื™ืœ ืึธื“ืขืจ ืกื˜ืึทื ื“ืึทืœืึธื ืข ืคืึธืจืขื, ื“ื™ืคึผืขื ื“ื™ื ื’ ืื•ื™ืฃ ื“ื™ ื•ื•ืึธื’ ืคื•ืŸ ื“ื™ ืคึผืจื•ื™ืขืงื˜ ืื•ืŸ ื“ื™ ื‘ืื“ืขืจืคืขื ื™ืฉืŸ ืคึฟืึทืจ ืคืึธืจืฉื˜ืขืœื•ื ื’ ืื•ืŸ ืฉื•ืœื“ ื˜ืึธืœืขืจืึทื ืฅ. ืงื™ื‘ืึทื ืึท ืื™ื– ื’ืขื ื™ืฆื˜ ื•ื•ื™ ื“ื™ ื’ืจืึทืคื™ืงืึทืœ ืฆื•ื‘ื™ื ื“.

ื•ื•ืขืŸ ื ื™ืฆืŸ Fluentd ืžื™ื˜ ืึท ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจื™ื ื’ ืคึผืœื•ื’ื™ืŸ, ืžื™ืจ ื’ืขืคึผืœืึธื ื˜ืขืจื˜ ืึท ืกื™ื˜ื•ืึทืฆื™ืข ื•ื•ื• ืขื˜ืœืขื›ืข ื“ืึธืงื•ืžืขื ื˜ืŸ ืื™ืŸ ElasticSearch ื”ืึธื‘ืŸ ืคึผื•ื ืงื˜ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืื™ื ื”ืึทืœื˜ ืื•ืŸ ื“ื™ืคืขืจื“ ื‘ืœื•ื™ื– ืื™ืŸ ื“ื™ ื™ื“ืขื ื˜ื™ืคื™ืขืจ. ืื™ืจ ืงืขื ืขืŸ ื‘ืึทืฉื˜ืขื˜ื™ืงืŸ ืึทื– ื“ืึธืก ืื™ื– ืึท ืึธื ื–ืึธื’ ื™ื‘ืขืจื›ืึทื–ืขืจื•ื ื’ ื ื™ืฆืŸ ื“ื™ Nginx ืงืœืึธืฅ ื•ื•ื™ ืึท ื‘ื™ื™ึทืฉืคึผื™ืœ. ืื™ืŸ ื“ื™ ืงืœืึธืฅ ื˜ืขืงืข, ื“ืขื ืึธื ื–ืึธื’ ื™ื’ื–ื™ืกืฅ ืื™ืŸ ืึท ืื™ื™ืŸ ืงืึธืคึผื™ืข:

127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -

ืึธื‘ืขืจ, ืขืก ื–ืขื ืขืŸ ืขื˜ืœืขื›ืข ื“ืึธืงื•ืžืขื ื˜ืŸ ืื™ืŸ ElasticSearch ื•ื•ืึธืก ืึทื ื˜ื”ืึทืœื˜ืŸ ื“ืขื ืึธื ื–ืึธื’:

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "HgGl_nIBR8C-2_33RlQV",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

{
  "_index": "test-custom-prod-example-2020.01.02",
  "_type": "_doc",
  "_id": "IgGm_nIBR8C-2_33e2ST",
  "_version": 1,
  "_score": 0,
  "_source": {
    "service": "test-custom-prod-example",
    "container_name": "nginx",
    "namespace": "test-prod",
    "@timestamp": "2020-01-14T05:29:47.599052886 00:00",
    "log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00  0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
    "tag": "custom-log"
  }
}

ื“ืขืจืฆื•, ืขืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ืžืขืจ ื•ื•ื™ ืฆื•ื•ื™ื™ ืจืขืคึผืึทื˜ื™ืฉืึทื ื–.

ื‘ืฉืขืช ืคื™ืงืกื™ืจ ื“ืขื ืคึผืจืึธื‘ืœืขื ืื™ืŸ ื“ื™ Fluentd ืœืึธื’ืก, ืื™ืจ ืงืขื ืขืŸ ื–ืขืŸ ืึท ื’ืจื•ื™ืก ื ื•ืžืขืจ ืคื•ืŸ ื•ื•ืึธืจื ื™ื ื’ื– ืžื™ื˜ ื“ื™ ืคืืœื’ืขื ื“ืข ืื™ื ื”ืึทืœื˜:

2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"

ื“ื™ ื•ื•ืึธืจื ื™ื ื’ื– ืคืึทืœืŸ ื•ื•ืขืŸ ElasticSearch ืงืขืŸ ื ื™ืฉื˜ ืฆื•ืจื™ืงืงื•ืžืขืŸ ืึท ืขื ื˜ืคืขืจ ืฆื• ืึท ื‘ืงืฉื” ืื™ืŸ ื“ืขืจ ืฆื™ื™ื˜ ืกืคึผืขืกื™ืคื™ืขื“ ื“ื•ืจืš ื“ื™ request_timeout ืคึผืึทืจืึทืžืขื˜ืขืจ, ื•ื•ืึธืก ืื™ื– ื•ื•ืึธืก ื“ื™ ืคืึธืจื•ื•ืขืจื“ื™ื“ ื‘ืึทืคืขืจ ืคืจืึทื’ืžืขื ื˜ ืงืขื ืขืŸ ื ื™ื˜ ื–ื™ื™ืŸ ืงืœื™ืจื“. ื ืึธืš ื“ืขื, Fluentd ืคืจื•ื•ื•ื˜ ืฆื• ืฉื™ืงืŸ ื“ื™ ื‘ืึทืคืขืจ ืคืจืึทื’ืžืขื ื˜ ืฆื• ElasticSearch ื•ื•ื™ื“ืขืจ ืื•ืŸ ื ืึธืš ืึท ืึทืจื‘ื™ื˜ืจืึทืจื™ืฉ ื ื•ืžืขืจ ืคื•ืŸ ืคืจื•ื•ื•ืŸ, ื“ื™ ืึธืคึผืขืจืึทืฆื™ืข ืงืึทืžืคึผืœื™ืฅ ื”ืฆืœื—ื”:

2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce" 
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2" 
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b" 
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"

ืึธื‘ืขืจ, ElasticSearch ื‘ืื”ืื ื“ืœื˜ ื™ืขื“ืขืจ ืคื•ืŸ ื“ื™ ื˜ืจืึทื ืกืคืขืจื“ ื‘ืึทืคืขืจ ืคืจืึทื’ืžืึทื ืฅ ื•ื•ื™ ื™ื™ื ืฆื™ืง ืื•ืŸ ืึทืกื™ื™ื ื– ื–ื™ื™ ื™ื™ื ืฆื™ืง _id ืคืขืœื“ ื•ื•ืึทืœื•ืขืก ื‘ืขืฉืึทืก ื™ื ื“ืขืงืกื™ื ื’. ื“ืึธืก ืื™ื– ื•ื•ื™ ืงืึธืคึผื™ืขืก ืคื•ืŸ ืึทืจื˜ื™ืงืœืขืŸ ื“ืขืจืฉื™ื™ึทื ืขืŸ.

ืื™ืŸ ืงื™ื‘ืึทื ืึท ืขืก ืงื•ืงื˜ ื•ื•ื™ ื“ืึธืก:

ืคืœื•ืขื ื˜ื“: ืคืืจื•ื•ืืก ืขืก ืื™ื– ื•ื•ื™ื›ื˜ื™ืง ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจ

ื“ื™ ืœื™ื™ื–ื•ื ื’

ืขืก ื–ืขื ืขืŸ ืขื˜ืœืขื›ืข ืึธืคึผืฆื™ืขืก ืฆื• ืกืึธืœื•ื•ืข ื“ืขื ืคึผืจืึธื‘ืœืขื. ืื™ื™ื ืขืจ ืคื•ืŸ ื–ื™ื™ ืื™ื– ื“ื™ ืžืขืงืึทื ื™ื–ืึทื ื’ืขื‘ื•ื™ื˜ ืื™ืŸ ื“ื™ ื’ืœืึทื˜-ืคึผืœื•ื’ื™ืŸ-elasticsearch ืคึผืœื•ื’ื™ืŸ ืคึฟืึทืจ ื“ื–ืฉืขื ืขืจื™ื™ื˜ื™ื ื’ ืึท ื™ื™ื ืฆื™ืง ื”ืึทืฉ ืคึฟืึทืจ ื™ืขื“ืขืจ ื“ืึธืงื•ืžืขื ื˜. ืื•ื™ื‘ ืื™ืจ ื ื•ืฆืŸ ื“ืขื ืžืขืงืึทื ื™ื–ืึทื, ElasticSearch ื•ื•ืขื˜ ื“ืขืจืงืขื ืขืŸ ืจืขืคึผืึทื˜ื™ืฉืึทื ื– ืื™ืŸ ื“ื™ ืคืึธืจื•ื•ืขืจื“ื™ื ื’ ื‘ื™ื ืข ืื•ืŸ ืคืึทืจืžื™ื™ึทื“ืŸ ื“ื•ืคึผืœื™ืงืึทื˜ ื“ืึธืงื•ืžืขื ื˜ืŸ. ืื‘ืขืจ ืžื™ืจ ืžื•ื–ืŸ ื ืขืžืขืŸ ืื™ืŸ ื—ืฉื‘ื•ืŸ ืึทื– ื“ืขื ืื•ืคึฟืŸ ืคื•ืŸ ืกืึทืœื•ื•ื™ื ื’ ื“ื™ ืคึผืจืึธื‘ืœืขื ืื™ื– ืกื˜ืจืึทื’ืึทืœื™ื ื’ ืžื™ื˜ ื“ื™ ื•ื™ืกืคืึธืจืฉื•ื ื’ ืื•ืŸ ื˜ื•ื˜ ื ื™ืฉื˜ ืขืœื™ืžื™ื ื™ืจืŸ ื“ืขื ื˜ืขื•ืช ืžื™ื˜ ืึท ืคืขืœืŸ ืคื•ืŸ ื˜ื™ื™ืžืึทื•ื˜, ืึทื–ื•ื™ ืžื™ืจ ืคืืจืœืื–ืŸ ื–ื™ื™ึทืŸ ื ื•ืฆืŸ.

ืžื™ืจ ื ื•ืฆืŸ ืึท ื‘ืึทืคืขืจื™ื ื’ ืคึผืœื•ื’ื™ืŸ ืื•ื™ืฃ ื“ื™ Fluentd ืจืขื–ื•ืœื˜ืึทื˜ ืฆื• ืคืึทืจืžื™ื™ึทื“ืŸ ืงืœืึธืฅ ืึธื ื•ื•ืขืจ ืื™ืŸ ืคืึทืœ ืคื•ืŸ ืงื•ืจืฅ-ื˜ืขืจืžื™ืŸ ื ืขืฅ ืคึผืจืึธื‘ืœืขืžืก ืึธื“ืขืจ ื’ืขื•ื•ืืงืกืŸ ืœืึธื’ื™ื ื’ ื™ื ื˜ืขื ืกื™ื˜ื™. ืื•ื™ื‘ ืคึฟืึทืจ ืขื˜ืœืขื›ืข ืกื™ื‘ื” ElasticSearch ืื™ื– ื ื™ื˜ ื‘ื™ื›ื•ืœืช ืฆื• ื˜ื™ื™ืงืขืฃ ืฉืจื™ื™ึทื‘ืŸ ืึท ื“ืึธืงื•ืžืขื ื˜ ืฆื• ื“ื™ ืื™ื ื“ืขืงืก, ื“ืขืจ ื“ืึธืงื•ืžืขื ื˜ ืื™ื– ืงื™ื™ ืื•ืŸ ืกื˜ืึธืจื“ ืื•ื™ืฃ ื“ื™ืกืง. ื“ืขืจื™ื‘ืขืจ, ืื™ืŸ ืื•ื ื“ื–ืขืจ ืคืึทืœ, ืื™ืŸ ืกื“ืจ ืฆื• ืขืœื™ืžื™ื ื™ืจืŸ ื“ื™ ืžืงื•ืจ ืคื•ืŸ ื“ื™ ืคึผืจืึธื‘ืœืขื ื•ื•ืึธืก ืคื™ืจื˜ ืฆื• ื“ื™ ื˜ืขื•ืช ื“ื™ืกืงืจื™ื™ื‘ื“ ืื•ื™ื‘ืŸ, ืขืก ืื™ื– ื ื™ื™ื˜ื™ืง ืฆื• ืฉื˜ืขืœืŸ ื“ื™ ืจื™ื›ื˜ื™ืง ื•ื•ืึทืœื•ืขืก ืคึฟืึทืจ ื“ื™ ื‘ืึทืคืขืจื™ื ื’ ืคึผืึทืจืึทืžืขื˜ืขืจืก, ืื™ืŸ ื•ื•ืึธืก ื“ื™ Fluentd ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจ ื•ื•ืขื˜ ื–ื™ื™ืŸ ื’ืขื ื•ื’ ื’ืจื™ื™ืก ืื•ืŸ ืื™ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืฆื™ื™ึทื˜ ืคื™ืจืŸ ืฆื• ื–ื™ื™ืŸ ืงืœื™ืจื“ ืื™ืŸ ื“ื™ ืึทืœืึทื˜ื™ื“ ืฆื™ื™ึทื˜.

ืขืก ืื™ื– ื›ื“ืื™ ืฆื• ื‘ืืžืขืจืงืŸ ืึทื– ื“ื™ ื•ื•ืึทืœื•ืขืก ืคื•ืŸ ื“ื™ ืคึผืึทืจืึทืžืขื˜ืขืจืก ื“ื™ืกืงืึทืกื˜ ืื•ื ื˜ืŸ ื–ืขื ืขืŸ ื™ื—ื™ื“ ืื™ืŸ ื™ืขื“ืขืจ ืกืคึผืขืฆื™ืคื™ืฉ ืคืึทืœ ืคื•ืŸ ื ื™ืฆืŸ ื‘ืึทืคืขืจื™ื ื’ ืื™ืŸ ืจืขื–ื•ืœื˜ืึทื˜ ืคึผืœื•ื’ื™ื ืก, ื•ื•ื™ื™ึทืœ ื–ื™ื™ ืึธืคืขื ื’ืขืŸ ืื•ื™ืฃ ืคื™ืœืข ืกื™ื‘ื•ืช: ื“ื™ ื™ื ื˜ืขื ืกื™ื˜ื™ ืคื•ืŸ ืฉืจื™ื™ื‘ืŸ ืึทืจื˜ื™ืงืœืขืŸ ืฆื• ื“ื™ ืงืœืึธืฅ ื“ื•ืจืš ืกืขืจื•ื•ื™ืกืขืก, ื“ื™ืกืง ืกื™ืกื˜ืขื ืคืึธืจืฉื˜ืขืœื•ื ื’, ื ืขืฅ ืงืึทื ืึทืœ ืžืึทืกืข ืื•ืŸ ื–ื™ื™ึทืŸ ื‘ืึทื ื“ื•ื•ื™ื“ื˜. ื“ืขืจื™ื‘ืขืจ, ื›ึผื“ื™ ืฆื• ื‘ืึทืงื•ืžืขืŸ ื‘ืึทืคืขืจ ืกืขื˜ื˜ื™ื ื’ืก ื•ื•ืึธืก ื–ืขื ืขืŸ ืคึผืึทืกื™ืง ืคึฟืึทืจ ื™ืขื“ืขืจ ื™ื—ื™ื“ ืคืึทืœ, ืึธื‘ืขืจ ื ื™ืฉื˜ ื™ื‘ืขืจื™ืง, ื•ื™ืกืžื™ื™ื“ืŸ ืœืึทื ื’ ืึธื ืคึฟืจืขื’ืŸ ื‘ืœื™ื™ื ื“ืœื™, ืื™ืจ ืงืขื ืขืŸ ื ื•ืฆืŸ ื“ื™ ื“ื™ื‘ืึทื’ื™ื ื’ ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข ื•ื•ืึธืก Fluentd ืฉืจื™ื™ื‘ื˜ ืฆื• ื–ื™ื™ืŸ ืงืœืึธืฅ ื‘ืขืฉืึทืก ืึธืคึผืขืจืึทืฆื™ืข ืื•ืŸ ืœืขืคื™ืขืจืขืš ื’ืขืฉื•ื•ื™ื ื“ ื‘ืึทืงื•ืžืขืŸ ื“ื™ ืจื™ื›ื˜ื™ืง ื•ื•ืึทืœื•ืขืก.

ืื™ืŸ ื“ืขืจ ืฆื™ื™ื˜ ื•ื•ืขืŸ ื“ื™ ืคึผืจืึธื‘ืœืขื ืื™ื– ืจืขืงืึธืจื“ืขื“, ื“ื™ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ื’ืขืงื•ืงื˜ ื•ื•ื™ ื“ืึธืก:

 <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.test.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 8M
        queue_limit_length 8
        overflow_action block
      </buffer>

ื•ื•ืขืŸ ืกืึทืœื•ื•ื™ื ื’ ื“ื™ ืคึผืจืึธื‘ืœืขื, ื“ื™ ื•ื•ืึทืœื•ืขืก ืคื•ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืคึผืึทืจืึทืžืขื˜ืขืจืก ื–ืขื ืขืŸ ืžืึทื ื™ื•ืึทืœื™ ืื•ื™ืกื’ืขืงืœื™ื‘ืŸ:
chunk_limit_size - ื“ื™ ื’ืจื™ื™ืก ืคื•ืŸ ื“ื™ ื˜ืฉืึทื ื’ืงืก ืื™ืŸ ื•ื•ืึธืก ืึทืจื˜ื™ืงืœืขืŸ ืื™ืŸ ื“ื™ ื‘ืึทืคืขืจ ื–ืขื ืขืŸ ืฆืขื˜ื™ื™ืœื˜.

  • flush_interval - ืฆื™ื™ื˜ ืžืขื”ืึทืœืขืš ื ืึธืš ื•ื•ืึธืก ื“ื™ ื‘ืึทืคืขืจ ืื™ื– ืงืœื™ืจื“.
  • queue_limit_length - ื“ื™ ืžืึทืงืกื™ืžื•ื ื ื•ืžืขืจ ืคื•ืŸ ื˜ืฉืึทื ื’ืงืก ืื™ืŸ ื“ืขืจ ืจื™ื™.
  • request_timeout ืื™ื– ื“ื™ ืฆื™ื™ื˜ ืคึฟืึทืจ ื•ื•ืึธืก ื“ื™ ืงืฉืจ ืฆื•ื•ื™ืฉืŸ Fluentd ืื•ืŸ ElasticSearch ืื™ื– ื’ืขื’ืจื™ื ื“ืขื˜.

ื“ื™ ื’ืึทื ืฅ ื‘ืึทืคืขืจ ื’ืจื™ื™ืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ืงืึทืœืงื™ืึทืœื™ื™ื˜ื™ื“ ื“ื•ืจืš ืžืึทืœื˜ืึทืคึผืœื™ื™ื™ื ื’ ื“ื™ ืคึผืึทืจืึทืžืขื˜ืขืจืก queue_limit_length ืื•ืŸ chunk_limit_size, ื•ื•ืึธืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ื™ื ื˜ืขืจืคึผืจืึทื˜ืึทื“ ื•ื•ื™ "ื“ื™ ืžืึทืงืกื™ืžื•ื ื ื•ืžืขืจ ืคื•ืŸ ื˜ืฉืึทื ื’ืงืก ืื™ืŸ ื“ืขืจ ืจื™ื™, ื™ืขื“ืขืจ ืคื•ืŸ ื•ื•ืึธืก ื”ืื˜ ืึท ื’ืขื’ืขื‘ืŸ ื’ืจื™ื™ืก." ืื•ื™ื‘ ื“ื™ ื‘ืึทืคืขืจ ื’ืจื™ื™ืก ืื™ื– ื ื™ื˜ ื’ืขื ื•ื’ื™ืง, ื“ื™ ืคืืœื’ืขื ื“ืข ื•ื•ืืจืขื ื•ื ื’ ื•ื•ืขื˜ ื“ืขืจืฉื™ื™ึทื ืขืŸ ืื™ืŸ ื“ื™ ืœืึธื’ืก:

2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:block

ืขืก ืžื™ื˜ืœ ืึทื– ื“ืขืจ ื‘ืึทืคืขืจ ื˜ื•ื˜ ื ื™ืฉื˜ ื”ืึธื‘ืŸ ืฆื™ื™ื˜ ืฆื• ื–ื™ื™ืŸ ืงืœื™ืจื“ ืื™ืŸ ื“ื™ ืึทืœืึทื˜ื™ื“ ืฆื™ื™ื˜ ืื•ืŸ ื“ื™ ื“ืึทื˜ืŸ ื•ื•ืึธืก ืึทืจื™ื™ึทืŸ ื“ื™ ืคื•ืœ ื‘ืึทืคืขืจ ื–ืขื ืขืŸ ืืคื’ืขืฉื˜ืขืœื˜, ื•ื•ืึธืก ื•ื•ืขื˜ ืคื™ืจืŸ ืฆื• ื“ื™ ืึธื ื•ื•ืขืจ ืคื•ืŸ ื˜ื™ื™ืœ ืคื•ืŸ ื“ื™ ืœืึธื’ืก.

ืื™ืจ ืงืขื ืขืŸ ืคืึทืจื’ืจืขืกืขืจืŸ ื“ื™ ื‘ืึทืคืขืจ ืื™ืŸ ืฆื•ื•ื™ื™ ื•ื•ืขื’ืŸ: ื“ื•ืจืš ื™ื ืงืจื™ืกื™ื ื’ ืึธื“ืขืจ ื“ื™ ื’ืจื™ื™ืก ืคื•ืŸ ื™ืขื“ืขืจ ืฉื˜ื™ืง ืื™ืŸ ื“ืขืจ ืจื™ื™, ืึธื“ืขืจ ื“ื™ ื ื•ืžืขืจ ืคื•ืŸ ื˜ืฉืึทื ื’ืงืก ื•ื•ืึธืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ืื™ืŸ ื“ืขืจ ืจื™ื™.

ืื•ื™ื‘ ืื™ืจ ืฉื˜ืขืœืŸ ื“ื™ ื˜ืฉืึทื ืง ื’ืจื™ื™ืก chunk_limit_size ืฆื• ืžืขืจ ื•ื•ื™ 32 ืžืขื’ืื‘ื™ื™ื˜, ElasticSeacrh ื•ื•ืขื˜ ื ื™ืฉื˜ ืึธื ื ืขืžืขืŸ ืขืก, ื•ื•ื™ื™ึทืœ ื“ื™ ื™ื ืงืึทืžื™ื ื’ ืคึผืึทืงืึทื˜ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืฆื• ื’ืจื•ื™ืก. ื“ืขืจื™ื‘ืขืจ, ืื•ื™ื‘ ืื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืคืึทืจื’ืจืขืกืขืจืŸ ื“ื™ ื‘ืึทืคืขืจ ื•ื•ื™ื™ึทื˜ืขืจ, ืขืก ืื™ื– ื‘ืขืกืขืจ ืฆื• ืคืึทืจื’ืจืขืกืขืจืŸ ื“ื™ ืžืึทืงืกื™ืžื•ื ืจื™ื™ ืœืขื ื’ queue_limit_length.

ื•ื•ืขืŸ ื“ืขืจ ื‘ืึทืคืขืจ ืกื˜ืึทืคึผืก ืึธื•ื•ื•ืขืจืคืœืึธื•ื™ื ื’ ืื•ืŸ ื‘ืœื•ื™ื– ื“ื™ ื˜ื™ื™ืžืึทื•ื˜ ื ื™ื˜ ื’ืขื ื•ื’ื™ืง ืึธื ื–ืึธื’ ื‘ืœื™ื™ื‘ื˜, ืื™ืจ ืงืขื ืขืŸ ืึธื ื”ื™ื™ื‘ืŸ ื™ื ืงืจื™ืกื™ื ื’ ื“ื™ request_timeout ืคึผืึทืจืึทืžืขื˜ืขืจ. ืึธื‘ืขืจ, ืื•ื™ื‘ ืื™ืจ ืฉื˜ืขืœืŸ ื“ื™ ื•ื•ืขืจื˜ ืฆื• ืžืขืจ ื•ื•ื™ 20 ืกืขืงื•ื ื“ืขืก, ื“ื™ ืคืืœื’ืขื ื“ืข ื•ื•ืึธืจื ื™ื ื’ื– ื•ื•ืขื˜ ืึธื ื”ื™ื™ื‘ืŸ ืฆื• ื“ืขืจืฉื™ื™ึทื ืขืŸ ืื™ืŸ ื“ื™ ืคืœื•ืขื ื˜ื“ ืœืึธื’ืก:

2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" 

ื“ืขืจ ืึธื ื–ืึธื’ ื˜ื•ื˜ ื ื™ืฉื˜ ื•ื•ื™ืจืงืŸ ื“ื™ ืึธืคึผืขืจืึทืฆื™ืข ืคื•ืŸ โ€‹โ€‹โ€‹โ€‹ื“ื™ ืกื™ืกื˜ืขื ืื™ืŸ ืงื™ื™ืŸ ื•ื•ืขื’ ืื•ืŸ ืžื™ื˜ืœ ืึทื– ื“ื™ ื‘ืึทืคืขืจ ืคืœืึทืฉ ืฆื™ื™ื˜ ื’ืขื ื•ืžืขืŸ ืžืขืจ ื•ื•ื™ ื‘ืึทืฉื˜ื™ืžื˜ ื“ื•ืจืš ื“ื™ slow_flush_log_threshold ืคึผืึทืจืึทืžืขื˜ืขืจ. ื“ืึธืก ืื™ื– ื“ื™ื‘ืึทื’ื™ื ื’ ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข ืื•ืŸ ืžื™ืจ ื ื•ืฆืŸ ืขืก ื•ื•ืขืŸ ื˜ืฉื•ื–ื™ื ื’ ื“ื™ ื•ื•ืขืจื˜ ืคื•ืŸ ื“ื™ request_timeout ืคึผืึทืจืึทืžืขื˜ืขืจ.

ื“ืขืจ ื’ืขื ืขืจืึทืœื™ื–ืขื“ ืกืขืœืขืงืฆื™ืข ืึทืœื’ืขืจื™ื“ืึทื ืื™ื– ื•ื•ื™ ื’ื™ื™ื˜:

  1. ืฉื˜ืขืœืŸ request_timeout ืฆื• ืึท ื•ื•ืขืจื˜ ื’ืขืจืึทื ื˜ื™ื“ ืฆื• ื–ื™ื™ืŸ ื’ืจืขืกืขืจ ื•ื•ื™ ื ื™ื™ื˜ื™ืง (ื”ื•ื ื“ืขืจื˜ืขืจ ืกืขืงื•ื ื“ืขืก). ื‘ืขืฉืึทืก ืกืขื˜ืึทืคึผ, ื“ื™ ื”ื•ื™ืคึผื˜ ืงืจื™ื˜ืขืจื™ืึธืŸ ืคึฟืึทืจ ื“ื™ ืจื™ื›ื˜ื™ืง ื‘ืึทืฉื˜ืขื˜ื™ืงืŸ ืคื•ืŸ ื“ืขื ืคึผืึทืจืึทืžืขื˜ืขืจ ืื™ื– ื“ื™ ื“ื™ืกืึทืคึผื™ืจืึทื ืก ืคื•ืŸ ื•ื•ืึธืจื ื™ื ื’ื– ืคึฟืึทืจ ืึท ืคืขืœืŸ ืคื•ืŸ ื˜ื™ื™ืžืึทื•ื˜.
  2. ื•ื•ืึทืจื˜ืŸ ืคึฟืึทืจ ืึทืจื˜ื™ืงืœืขืŸ ื•ื•ืขื’ืŸ ื™ืงืกื™ื“ ื“ื™ slow_flush_log_threshold ืฉื•ื•ืขืœ. ื“ื™ ื•ื•ืืจืขื ื•ื ื’ ื˜ืขืงืกื˜ ืื™ืŸ ื“ื™ elapsed_time ืคืขืœื“ ื•ื•ืขื˜ ื•ื•ื™ื™ึทื–ืŸ ื“ื™ ืคืึทืงื˜ื™ืฉ ืฆื™ื™ื˜ ื•ื•ืขืŸ ื“ื™ ื‘ืึทืคืขืจ ืื™ื– ืงืœื™ืจื“.
  3. ืฉื˜ืขืœืŸ request_timeout ืฆื• ืึท ื•ื•ืขืจื˜ ื’ืจืขืกืขืจ ื•ื•ื™ ื“ื™ ืžืึทืงืกื™ืžื•ื elapsed_time ื•ื•ืขืจื˜ ื‘ืืงื•ืžืขืŸ ื‘ืขืฉืึทืก ื“ื™ ืึธื‘ืกืขืจื•ื•ืึทืฆื™ืข ืฆื™ื™ึทื˜. ืžื™ืจ ืจืขื›ืขื ืขืŸ ื“ื™ request_timeout ื•ื•ืขืจื˜ ื•ื•ื™ elapsed_time + 50%.
  4. ืฆื• ื‘ืึทื–ื™ื™ึทื˜ื™ืงืŸ ื•ื•ืึธืจื ื™ื ื’ื– ื•ื•ืขื’ืŸ ืœืึทื ื’ ื‘ืึทืคืขืจ ืคืœื•ืฉืขืก ืคื•ืŸ ื“ื™ ืงืœืึธืฅ, ืื™ืจ ืงืขื ืขืŸ ื›ืึทืคึผืŸ ื“ื™ ื•ื•ืขืจื˜ ืคื•ืŸ slow_flush_log_threshold. ืžื™ืจ ืจืขื›ืขื ืขืŸ ื“ืขื ื•ื•ืขืจื˜ ื•ื•ื™ elapsed_time + 25%.

ื“ื™ ืœืขืฆื˜ ื•ื•ืึทืœื•ืขืก ืคื•ืŸ ื“ื™ ืคึผืึทืจืึทืžืขื˜ืขืจืก, ื•ื•ื™ ื“ืขืจืžืื ื˜ ืคืจื™ืขืจ, ื–ืขื ืขืŸ ื‘ืืงื•ืžืขืŸ ื™ื ื“ื™ื•ื•ื™ื“ื–ืฉื•ืึทืœื™ ืคึฟืึทืจ ื™ืขื“ืขืจ ืคืึทืœ. ื“ื•ืจืš ื ืึธื›ืคืึธืœื’ืŸ ื“ื™ ืึทืœื’ืขืจื™ื“ืึทื ืื•ื™ื‘ืŸ, ืžื™ืจ ื–ืขื ืขืŸ ื’ืขืจืึทื ื˜ื™ื“ ืฆื• ืขืœื™ืžื™ื ื™ืจืŸ ื“ื™ ื˜ืขื•ืช ื•ื•ืึธืก ืคื™ืจื˜ ืฆื• ืจื™ืคึผื™ื˜ื™ื“ ืึทืจื˜ื™ืงืœืขืŸ.

ื“ื™ ื˜ื™ืฉ ืื•ื ื˜ืŸ ื•ื•ื™ื™ื–ื˜ ื•ื•ื™ ื“ื™ ื ื•ืžืขืจ ืคื•ืŸ ืขืจืจืึธืจืก ืคึผืขืจ ื˜ืึธื’, ื•ื•ืึธืก ืคื™ืจืŸ ืฆื• ื“ื•ืคึผืœื™ืงื™ื™ืฉืึทืŸ ืคื•ืŸ ืึทืจื˜ื™ืงืœืขืŸ, ืขื ื“ืขืจื•ื ื’ืขืŸ ืื™ืŸ ื“ืขื ืคึผืจืึธืฆืขืก ืคื•ืŸ ืกืขืœืขืงื˜ื™ื ื’ ื“ื™ ื•ื•ืึทืœื•ืขืก ืคื•ืŸ ื“ื™ ืื•ื™ื‘ืŸ ื“ื™ืกืงืจื™ื™ื‘ื“ ืคึผืึทืจืึทืžืขื˜ืขืจืก:

ื ืึธื“ืข-1
ื ืึธื“ืข-2
ื ืึธื“ืข-3
ื ืึธื“ืข-4

ืื™ื™ื“ืขืจ ื ืึธืš
ืื™ื™ื“ืขืจ ื ืึธืš
ืื™ื™ื“ืขืจ ื ืึธืš
ืื™ื™ื“ืขืจ ื ืึธืš

ื ื™ื˜ ืึทื ื“ืขืจืฉ ืฆื• ื•ื™ืกืฉืขืคึผืŸ ื“ื™ ื‘ืึทืคืขืจ
1749/2
694/2
47/0
1121/2

ื•ื•ื™ื“ืขืจ ื’ืขืจืึธื˜ืŸ
410/2
205/1
24/0
241/2

ืขืก ืื™ื– ืื•ื™ืš ื›ื“ืื™ ืฆื• ื‘ืืžืขืจืงืŸ ืึทื– ื“ื™ ืจื™ื–ืึทืœื˜ื™ื ื’ ืกืขื˜ื˜ื™ื ื’ืก ืงืขืŸ ืคืึทืจืœื™ืจืŸ ื–ื™ื™ืขืจ ืฉื™ื™ื›ื•ืช ื•ื•ืขืŸ ื“ื™ ืคึผืจื•ื™ืขืงื˜ ื•ื•ืืงืกื˜ ืื•ืŸ, ืึทืงืึธืจื“ื™ื ื’ืœื™, ื“ื™ ื ื•ืžืขืจ ืคื•ืŸ ืœืึธื’ืก ื™ื ืงืจื™ืกื™ื–. ื“ืขืจ ืขืจืฉื˜ื™ืง ืฆื™ื™ื›ืŸ ืคื•ืŸ ื ื™ื˜ ื’ืขื ื•ื’ื™ืง ื˜ื™ื™ืžืึทื•ื˜ ืื™ื– ื“ื™ ืฆื•ืจื™ืงืงืขืจ ืคื•ืŸ ืึทืจื˜ื™ืงืœืขืŸ ื•ื•ืขื’ืŸ ืึท ืœืึทื ื’ ื‘ืึทืคืขืจ ื’ืœื™ื™ึทืš ืฆื• ื“ื™ Fluentd ืงืœืึธืฅ, ื“ืึธืก ืื™ื– ื™ืงืกื™ื“ ื“ื™ ืกืœืึธื•_ืคืœื•ืฉ_ืœืึธื’_ื˜ื”ืจืขืฉืึธืœื“ ืฉื•ื•ืขืœ. ืคึฟื•ืŸ ื“ืขื ืคื•ื ื˜ ืื•ื™ืฃ, ืขืก ืื™ื– ื ืึธืš ืึท ืงืœื™ื™ืŸ ื’ืจืขื ืขืฅ ืื™ื™ื“ืขืจ ื“ื™ ืจืขืงื•ื•ื™ืจืข_ื˜ื™ืžืขืึธื•ื˜ ืคึผืึทืจืึทืžืขื˜ืขืจ ืื™ื– ื™ืงืกื™ื“ื™ื“, ืึทื–ื•ื™ ืขืก ืื™ื– ื ื™ื™ื˜ื™ืง ืฆื• ืจื™ืกืคึผืึทื ื“ ืฆื• ื“ื™ ืึทืจื˜ื™ืงืœืขืŸ ืื™ืŸ ืึท ื‘ื™ื™ึทืฆื™ื™ึทื˜ื™ืง ืฉื˜ื™ื™ื’ืขืจ ืื•ืŸ ืื™ื‘ืขืจื—ื–ืจืŸ ื“ืขื ืคึผืจืึธืฆืขืก ืคื•ืŸ ืกืึทืœืขืงื˜ื™ื ื’ ื“ื™ ืึธืคึผื˜ื™ืžืึทืœ ืกืขื˜ื˜ื™ื ื’ืก ื“ื™ืกืงืจื™ื™ื‘ื“ ืื•ื™ื‘ืŸ.

ืกืึธืฃ

ืคื™ื™ืŸ-ื˜ื•ื ื™ื ื’ ื“ื™ Fluentd ืจืขื–ื•ืœื˜ืึทื˜ ื‘ืึทืคืขืจ ืื™ื– ืื™ื™ื ืขืจ ืคื•ืŸ ื“ื™ ื”ื•ื™ืคึผื˜ ืกื˜ืึทื’ืขืก ืคื•ืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื ื’ ื“ื™ EFK ืึธื ืœื™ื™ื’ืŸ, ื“ื™ื˜ืขืจืžืึทื ื™ื ื’ ื“ื™ ืคืขืกื˜ืงื™ื™ึทื˜ ืคื•ืŸ ื–ื™ื™ึทืŸ ืึธืคึผืขืจืึทืฆื™ืข ืื•ืŸ ื“ื™ ืจื™ื›ื˜ื™ืง ืคึผืœื™ื™ืกืžืึทื ื˜ ืคื•ืŸ ื“ืึธืงื•ืžืขื ื˜ืŸ ืื™ืŸ ื™ื ื“ืขืงืกื™ื–. ื‘ืึทื–ื™ืจื˜ ืื•ื™ืฃ ื“ื™ ื“ื™ืกืงืจื™ื™ื‘ื“ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืึทืœื’ืขืจื™ื“ืึทื, ืื™ืจ ืงืขื ืขืŸ ื–ื™ื™ืŸ ื–ื™ื›ืขืจ ืึทื– ืึทืœืข ืœืึธื’ืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ื’ืขืฉืจื™ื‘ืŸ ืฆื• ื“ื™ ElasticSearch ืื™ื ื“ืขืงืก ืื™ืŸ ื“ื™ ืจื™ื›ื˜ื™ืง ืกื“ืจ, ืึธืŸ ืจืขืคึผืึทื˜ื™ืฉืึทื ื– ืึธื“ืขืจ ืœืึธืกืกืขืก.

ืœื™ื™ืขื ืขืŸ ืื•ื™ืš ืื ื“ืขืจืข ืึทืจื˜ื™ืงืœืขืŸ ืื•ื™ืฃ ืื•ื ื“ื–ืขืจ ื‘ืœืึธื’:

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’