ืืืื ืคืึธืจืฉืืขืืื ื ืืื ืืืื ืขืจ ืคืื ืื ืฉืืืกื ืจืขืงืืืืจืขืืขื ืฅ ืืืขื ืืจืืขืื ืืื ืืจืืืก ืืึทืื. ืืื ืื ืืึทืื ืืึธืืืื ื ืึธืคึผืืืืืื ื ืืื ืกืืขืจืืึทื ืง, ืืืจ ืคึผืึธืืคึผืข ืึผืืขื ืึทืืข ืืจืึทื ืืึทืงืฉืึทื ื ืืื ืืื ืืืขืจ Hadoop-ืืืืืจื ืืึทืืึท ืงืืึธืื ืืื ืืขืจืืืขืจ ืืึทื ืืืขื ืืื ืืึทืงืข ืืจืืืก ืคืืึธืื ืคืื ืืื ืคึฟืึธืจืืึทืฆืืข. ืืขืืืืื ืืืขื, ืืืจ ืืขื ืขื ืฉืืขื ืืืง ืงืืงื ืคึฟืึทืจ ืืืขืื ืฆื ืคึฟืึทืจืืขืกืขืจื ืคืึธืจืฉืืขืืื ื, ืืื ืืืฆื ืืืจ ืืืขืื ืฆื ืืึธืื ืืืจ ืืื ืืืจ ืืขืจืืื ืฆื ืคึผืึทืืฉ RegionServer HBase ืืื ืื HDFS ืงืืืขื ื, ืืึทื ืง ืฆื ืืืึธืก ืืืจ ืืขื ืขื ืืืืืืช ืฆื ืืืืืืืืง ืคืึทืจืืจืขืกืขืจื ืื ืืืืงืืึทื ืคืื ืืืืขื ืขื ืึทืคึผืขืจืืืฉืึทื ื.
ืึธืืขืจ, ืืืืืขืจ ืืืจ ืืึทื ืืืืฃ ืฆื ืื ืขืกืึทื ืก ืคืื ืื ืืืคึผืจืืืืืึทื ืฅ, ืขืก ืืื ืืืขืจื ืฆื ืจืขืื ืืืขืื ืจืืกืืจืืงืฉืึทื ื ืืืึธืก, ืืื ืคึผืจืื ืฆืืคึผ, ืงืขื ืขื ื ืื ืืืื ืกืขืจืงืึทืืืืขื ืืื ืืืื ืืืจ ืืืฆื ืืืืฃ ืึท ืืื.
ืคืืจืืืืก HDD ืืื ืฉื ืขื ืจืึทื ืืึธื ืึทืงืกืขืก ืืืืขื ืขื ืืขื ืขื ืื ืงืึทืืคึผืึทืืึทืืึทื
ืืื ืืืจ ืืืืกื, HBase, ืืื ืคืืืข ืื ืืขืจืข ืืึทืืึทืืืืกืื, ืงืจืึธื ืืึทืื ืืื ืืืึทืงืก ืคืื ืขืืืขืืข ืืขื ืก ืคืื ืงืืืืืืื ืืื ืืจืืืก. ืืืจื ืคืขืืืงืืึทื ืขืก ืืื ืืืขืื 64 ืงื. ืืืฆื ืืึธืืืจ ืืืึทืืืฉืึทื ืึทื ืืืจ ืืึทืจืคึฟื ืฆื ืืึทืงืืืขื ืืืืื 100 ืืืืขืก ืืื ืืืจ ืืขืื HBase ืฆื ืืขืื ืืื ืื ืื ืืึทืื ืืื ืึท ืืืืขืจ ืฉืืืกื. ืืื ื ืื ืืืึธืง ืืจืืืก ืืื HFiles ืืื 64 ืงื, ืื ืืงืฉื ืืืขื ืืืื 640 ืืื ืืจืขืกืขืจ (ื ืึธืจ ืึท ืืื ืื!) ืืื ื ืืืืืง.
ืืขืจื ืึธื, ืืื ื ืื ืืงืฉื ืืืขื ืืืื ืืืจื HDFS ืืื ืืืื ืืขืืึทืืึทืืึท ืงืึทืืฉืื ื ืืขืงืึทื ืืืึทื ShortCircuitCache (ืืืึธืก ืึทืืึทืื ืืืจืขืงื ืึทืงืกืขืก ืฆื ืืขืงืขืก), ืืึธืก ืคืืจื ืฆื ืืืืขื ืขื ืฉืืื 1 ืื ืคืื ืื ืืืกืง. ืึธืืขืจ, ืืึธืก ืงืขื ืขื ืืืื ืึทืืืฉืึทืกืืื ืืื ืืขื ืคึผืึทืจืึทืืขืืขืจ ืืคืก.ืงืืืขื ื.ืืืืขื ืขื.ืงืืจืฅ-ืืืคืขืจ.ืืจืืืก ืืื ืืื ืคืืืข ืงืึทืกืขืก ืขืก ืืื ืืื ืขื ืฆื ืจืขืืืฆืืจื ืืขื ืืืขืจื, ืืืฉื ืฆื 126 ืงื.
ืืื ืก ืืึธืื ืืืจ ืืึธื ืืึธืก, ืึธืืขืจ ืืื ืึทืืืฉืึทื, ืืืขื ืืืจ ืึธื ืืืืื ืืืืขื ืขื ืืึทืื ืืืจื ืื ืืืฉืึทืืืึท ืึทืคึผื, ืึทืืึท ืืื ืคืึทื ืืงืฉืึทื ื ืืื FileChannel.read ืืื ืืขืื ืื ืึธืคึผืขืจืืืืื ื ืกืืกืืขื ืฆื ืืืืขื ืขื ืื ืกืคึผืขืกืืคืืขื ืกืืืข ืคืื โโืืึทืื, ืขืก ืืืืขื ื "ื ืึธืจ ืืื ืคืึทื" 2 ืืื ืืขืจ , ื.ื. 256 ืงื ืืื ืืื ืืืขืจ ืคืึทื. ืืึธืก ืืื ืืืืึทื Java ืืื ื ืืฉื ืึทื ืืจืื ื ืืืขื ืฆื ืฉืืขืื ืื FADV_RANDOM ืคืึธื ืฆื ืคืึทืจืืืึทืื ืืขื ื ืึทืืืจ.
ืืื ืึท ืจืขืืืืืึทื, ืฆื ืืึทืงืืืขื ืืื ืืืขืจ 100 ืืืืขืก, 2600 ืืื ืืขืจ ืืื ืืืืขื ืขื ืืื ืืขืจ ืื ืงืึทืคึผืืขืจ. ืขืก ืืืึธืื ืืืกืงืืืขื ืึทื ืื ืืืืืื ื ืืื ืงืืึธืจ ืืื ืืขืจ ืืึธื, ืืึธืื ืืื ืื ืจืขืืืฆืืจื ืื ืืจืืืก ืคืื ืื ืืืึธืง ืฆื ืงืืืึธืืืื, ืฉืืขืื ืื ืืขืจืืื ื ืคืึธื ืืื ืืึทืงืืืขื ืึท ืืจืืืก ืึทืงืกืขืืขืจืืืฉืึทื ืคืื ืืืืคืงืืขืจืื ื. ืืืขืจ ืื ืฆืจื ืืื ืึทื ืืืจื ืจืืืืกืื ื ืื ืืืึธืง ืืจืืืก ืืื 2 ืืื, ืืืจ ืืืื ืจืขืืืฆืืจื ืื ื ืืืขืจ ืคืื ืืืืขืก ืืืืขื ืขื ืคึผืขืจ ืึทืคึผืึทืจืึทื ืคืื ืฆืืึทื ืืื 2 ืืื.
ืขืืืขืืข ื ืืฅ ืคืื ืืึทืฉืืขืืืงื ืื FADV_RANDOM ืคืึธื ืงืขื ืขื ืืืื ืืืงืืืขื, ืึธืืขืจ ืืืืื ืืื ืืืื ืืืืื-ืคืึธืืขื ืืื ืืื ืึท ืืืึธืง ืืจืืืก ืคืื 128 ืงื, ืึธืืขืจ ืืึธืก ืืื ืึท ืืึทืงืกืืืื ืคืื ืึท ืคึผืึธืจ ืคืื ืืขื ืก ืคืื ืคึผืจืึธืฆืขื ื:
ืืขืกืฅ ืืขื ืขื ืืืจืืืขืงืึธืื ืืืืฃ 100 ืืขืงืขืก, ืืขืืขืจ 1 ืืืืืืืื ืืื ืืจืืืก ืืื ืืืื ืืืืฃ 10 ืืื.
ืืึธืืืจ ืจืขืืขื ืขื ืืืืฃ ืืืึธืก ืืืจ ืงืขื ืขื, ืืื ืคึผืจืื ืฆืืคึผ, ืจืขืืขื ืขื ืืื ืื ืืืืงืืึทื:
ืืื ืก ืืึธืื ืืืจ ืืืืขื ืขื ืคืื 10 ืืืกืงืก ืืื ืึท ืืืืงืืึทื ืคืื 280 ืืขืืืืืืื / ืกืขืง, ื.ื. 3 ืืืืืึธื ืืื 100 ืืืืขืก. ืืืขืจ ืืื ืืืจ ืืขืืขื ืงืขื, ืื ืืึทืื ืืืจ ืืึทืจืคึฟื ืืื 2600 ืืื ืืืืื ืืงืขืจ ืืื ืืืึธืก ืืื ืืืืขื ืขื. ืืืื, ืืืจ ืืืืื 3 ืืืืืึธื ืืืจื 2600 ืืื ืืึทืงืืืขื 1100 ืจืขืงืึธืจืืก ืคึผืขืจ ืกืขืงืื ืืข.
ืืขืคึผืจืขืกืืข, ืืื ื ืื ืขืก? ืึทื ืก ื ืึทืืืจ ืืจืึทืค - ืึทืงืกืขืก ืฆืืืจืื ืฆื ืืึทืื ืืืืฃ ืื ืืื - ืจืึทืืึทืจืืืึทืก ืคืื ืื ืืืึธืง ืืจืืืก. ืืึธืก ืืื ืื ืืฉืืืืช ืืืืื ืคืื ืืจืึทืค ืึทืงืกืขืก ืืื ืงืืื ืืึทืืึทืืืืก ืงืขื ืขื ื ืืฉื ืงืืืขืืฉื ืืขืจ ืืื ืืขืจ ืึทืืึท ืื ืึธืื.
ืืื ืึทืืื ืืึธื ืืึทืืึทืืืืกืื ืืขืจืืจืืืื ืคืื ืืขืืขืจ ืกืคึผืืื? ืฆื ืขื ืืคึฟืขืจื ืืขื ืงืฉืื, ืืึธืืืจ ืงืืงื ืืื ืืืึธืก ืืื ืืขืฉืขืขื ืืฉ ืืื ืื ืคืืืืขื ืืข ืืืื:
ืืึธ ืืืจ ืืขื ืึทื ืคึฟืึทืจ ืื ืขืจืฉืืขืจ ืืืกื ืืื ืื ืื ืืืืงืืึทื ืืื ืืึทืงืข ืืืขืื ืึท ืืืืื ื ืจืขืงืึธืจืืก ืคึผืขืจ ืกืขืงืื ืืข. ืึธืืขืจ, ืืืืึทืืขืจ, ืจืขืื ืฆื ืืขื ืคืึทืงื ืึทื ืคืื ืืขืจ ืืื ืืืืขื ืขื ืืื ืืขืืขืื, ืื ืืึทืื ืขื ืืก ืืื ืื ืืึทืฃ / ืงืึทืฉ ืคืื ืื ืึธืคึผืขืจืืืืื ื ืกืืกืืขื (ืืื ืืงืก) ืืื ืื ืืืืงืืึทื ืื ืงืจืืกืื ืฆื ืึท ืืขืจ ืืืึทืืืฉ 60 ืืืืื ื ืคึผืขืจ ืกืขืงืื ืืข
ืืขืจื ืึธื, ืืืจ ืืืขืื ืืึทื ืืืขื ืืื ืึทืงืกืขืืขืจืืืืื ื ืึทืงืกืขืก ืืืืื ืฆื ืื ืืึทืื ืืืึธืก ืืขื ืขื ืืื ืื ืึทืก ืงืึทืฉ ืึธืืขืจ ืืืื ืืื SSD / NVMe ืกืืึธืจืืืืฉ ืืขืืืืกืขืก ืืื ืคืึทืจืืืืึทืืืขื ืึทืงืกืขืก ืืืืงืืึทื.
ืืื ืืื ืืืขืจ ืคืึทื, ืืืจ ืืืขืื ืืืจืืคืืจื ืืขืกืฅ ืืืืฃ ืึท ืืึทื ืง ืคืื 4 ืกืขืจืืืขืจืก, ืืขืืขืจ ืคืื ืืืึธืก ืืื ืืืคืืืื ืืื ืืืื:
ืงืคึผื: Xeon E5-2680 v4 @ 2.40GHz 64 ืคึฟืขืืขื.
ืืึผืจืื: 730 ืืืืืืืื.
Java ืืืขืจืกืืข: 1.8.0_111
ืืื ืืึธ ืืขืจ ืฉืืืกื ืคืื ื ืืื ืื ืกืืืข ืคืื โโโโืืึทืื ืืื ืื ืืืฉื ืืืึธืก ืืึทืจืฃ ืืืื ืืืืขื ืขื. ืืขืจ ืคืึทืงื ืืื ืึทื ืืืื ืืืจ ืืืืขื ืขื ืืึทืื ืคืื ืึท ืืืฉ ืืืึธืก ืืื ืืขืืึทืืจืข ืืขืฉืืขืื ืืื ืื HBase ืงืึทืฉ, ืขืก ืืืขื ื ืืฉื ืืคืืื ืงืืืขื ืฆื ืืืืขื ืขื ืคึฟืื ืื ืึธืคึผืขืจืืืืื ื ืกืืกืืขื ืก ืืึทืฃ / ืงืึทืฉ. ืืืืึทื HBase ืืืจื ืคืขืืืงืืึทื ืึทืืึทืงืืืฅ 40% ืคืื ืืึผืจืื ืฆื ืึท ืกืืจืืงืืืจ ืืขืจืืคื BlockCache. ืืกืขื ืฉืึทืื, ืืึธืก ืืื ืึท ConcurrentHashMap, ืืื ืืขืจ ืฉืืืกื ืืื ืืขืจ ืืขืงืข ื ืึธืืขื + ืคืึธืืึธ ืคืื ืื ืืืึธืง, ืืื ืื ืืืขืจื ืืื ืื ืคืึทืงืืืฉ ืืึทืื ืืื ืืขื ืคืึธืืึธ.
ืืืื, ืืืขื ืืืืขื ืขื ืืืืื ืคืื ืืขื ืกืืจืืงืืืจ, ืืืจ
ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืื ืืื ืืืขืจ ืคืึทื, ืื ืืึทื ื ืคืื BlockCache ืืืืฃ ืืืื ืจืก ืืื ืืืขืื 12 ืืืืืืืื. ืืืจ ืืึทื ืืื ืฆืืืื ืจืก ืืืืฃ ืืืื ื ืึธืืข, ื.ื. 96 ืืืืืืืื ืืขื ืขื ืึทืืึทืงืืืืื ืคึฟืึทืจ BlockCache ืืืืฃ ืึทืืข ื ืึธืืื. ืืื ืขืก ืืขื ืขื ืคืืืข ืืึธื ืืขืจ ืืึทืื, ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืึธืื ืขืก ืืืื 4 ืืืฉื, 130 ืืงืืืืช ืืขืืขืจ, ืืื ืืืึธืก ืืขืงืขืก ืืขื ืขื 800 ืืขืืืืืืื ืืื ืืจืืืก, ืงืึทืืคึผืจืขืกื ืืืจื FAST_DIFF, ื.ืข. ืึท ืืึทื ืฅ ืคืื 410 ืืืืืืืื (ืืึธืก ืืื ืจืืื ืืึทืื, ื"ื ืึธื ืืขื ืืืขื ืืื ืืฉืืื ืื ืจืขืคึผืืึทืงืืืฉืึทื ืคืึทืงืืึธืจ).
ืืืื, BlockCache ืืื ืืืืื ืืืขืื 23% ืคืื ืื ืืึทื ืฅ ืืึทืื ืืึทื ื ืืื ืืึธืก ืืื ืคืื ื ืขืขื ืืขืจ ืฆื ืื ืคืึทืงืืืฉ ืื ืึธืื ืคืื ืืืึธืก ืืื ืืขืจืืคื BigData. ืืื ืืึธืก ืืื ืืื ืื ืฉืคึผืึทืก ืืืืื - ืืืืึทื ืืึธื, ืื ืืืืื ืืงืขืจืข ืงืึทืฉ ืืืฅ, ืื ืขืจืืขืจ ืื ืคืึธืจืฉืืขืืื ื. ื ืึธื ืึทืืข, ืืืื ืืืจ ืคืึทืจืคืืจื, ืืืจ ืืืขื ืืึธืื ืฆื ืืึธื ืึท ืคึผืืึทืฅ ืคืื ืึทืจืืขื - ื.ื. ืืืื ืึทืจืึธืคึผ ืฆื ืจืืคื ืกืืกืืขื ืคืึทื ืืงืฉืึทื ื. ืึธืืขืจ, ืืึธืก ืงืขื ื ืืฉื ืืืื ืึทืืืืืืื, ืึทืืื ืืึธืืืจ ืงืืงื ืืื ืึท ืืึธืจ ืึทื ืืขืจืฉ ืึทืกืคึผืขืงื - ืืืึธืก ืืึทืคึผืึทื ื ืืื ืื ืืึทืื ืืื ืื ืงืึทืฉ?
ืืื ืก ืคืึทืจืคึผืึธืฉืขืืขืจื ืื ืกืืืืึทืฆืืข ืืื ืืืขืจื ืขืืขื ืึทื ืืืจ ืืึธืื ืึท ืงืึทืฉ ืึทื ืืืืื ืคืืฅ 1 ืืืืคืขืฅ. ืืึธ ืืื ืึท ืืืืฉืคึผืื ืคืื ืืืึธืก ืืืขื ืคึผืึทืกืืจื ืืืขื ืืืจ ืคึผืจืืืืจื ืฆื ืึทืจืืขืื ืืื ืึท ืืึทืื ืืึทื ื 3 ืืื ืืจืขืกืขืจ ืืื ืื ืงืึทืฉ, ืืืจ ืืืขืื ืืึธืื ืฆื:
1. ืฉืืขืื ืืืึธืง 1 ืืื ืงืึทืฉ
2. ืึทืจืึธืคึผื ืขืืขื ืืืึธืง 1 ืคืื ืงืึทืฉ
3. ืฉืืขืื ืืืึธืง 2 ืืื ืงืึทืฉ
4. ืึทืจืึธืคึผื ืขืืขื ืืืึธืง 2 ืคืื ืงืึทืฉ
5. ืฉืืขืื ืืืึธืง 3 ืืื ืงืึทืฉ
5 ืึทืงืฉืึทื ื ืืขืขื ืืืงื! ืึธืืขืจ, ืื ืกืืืืึทืฆืืข ืงืขื ื ืืฉื ืืืื ืืขืจืืคึฟื ื ืึธืจืืึทื; ืืื ืคืึทืงื, ืืืจ ืคืึธืจืกืื ื HBase ืฆื ืืึธื ืึท ืคึผืืึทืฅ ืคืื ืืึธืจ ืึทืจืืืกืืขืืืึธืจืคื ืึทืจืืขื. ืขืก ืงืขืกืืืืขืจ ืืืืขื ื ืืึทืื ืคืื ืื ืึทืก ืงืึทืฉ, ืฉืืขืื ืขืก ืืื BlockCache, ื ืึธืจ ืฆื ืืืึทืจืคื ืขืก ืึผืืขื ืืืืื ืืืืึทื ืึท ื ืืึทืข ืืืง ืคืื ืืึทืื ืืื ืื ืืขืงืืืขื. ืื ืึทื ืึทืืืืฉืึทื ืืื ืื ืึธื ืืืื ืคืื ืื ืคึผืึธืกืื ืืืืืื ืื ืขืกืึทื ืก ืคืื ืืขื ืคึผืจืึธืืืขื - ืืืกื ืงืึทืืขืงืืขืจ ืืื ืึทืืืขืง ืคืื ืืืึธื, ืื ืึทืืืึธืกืคืขืจ ืืื ืืึทืืืฆืื ื ืึทืจืืืฃ, ืงืืืื ืืจืขืืึท ืืื ืืืืึทื ืืื ืืืืก ืฉืืืขืื ืืื ืืขืืื ื ืืืขืจืงืขืจื. ืืื ืืืจ ืขืก ืืขื ืืฉื ืืึทืงืข ืืึธื ื ืื ืืื ืขืก ืืืขื ืงืื ืืขืจ ืืขื ืขื ืืจืืืขืจืืง, ืึทืืื ืืืจ ืึธื ืืืืื ืฆื ืืจืึทืืื ืืืขืื ืืืึธืก ืืืจ ืงืขื ืขื ืืึธื ืืืขืื ืืื.
ืืืึธืก ืืืื ืืืจ ืฉืืขืื ื ืื ืึทืืข ืืืึทืงืก ืืื ืื ืงืึทืฉ, ืึธืืขืจ ืืืืื ืึท ืืืืขืจ ืคึผืจืึธืฆืขื ื ืคืื ืืื, ืึทืืื ืึทื ืื ืงืึทืฉ ืืืขื ื ืืฉื ืืืืคื? ืืึธืืืจ ืึธื ืืืืื ืืื ืคืฉืื ืึทืืื ื ืืืืื ืึท ืืืกื ืฉืืจืืช ืคืื ืงืึธื ืฆื ืื ืึธื ืืืื ืคืื ืื ืคืื ืงืฆืืข ืคึฟืึทืจ ืคึผืึทืืื ื ืืึทืื ืืื BlockCache:
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...
ืื ืคืื ื ืืึธ ืืื ืื ืคืืืืขื ืืข: ืคืึธืืึธ ืืื ืื ืฉืืขืืข ืคืื โโืื ืืืึธืง ืืื ืืขืจ ืืขืงืข ืืื ืื ืืขืฆืืข ืืืืืฉืึทืฅ ืืขื ืขื ืจืึทื ืืึทืืื ืืื ืืืืึทื ืื ืคืื ืื ืืขืจืืขืืืืื ืคืื 00 ืฆื 99. ืืขืจืืืขืจ, ืืืจ ืืืขืื ื ืึธืจ ืืึธืคึผืงืขื ืื ืืืึธืก ืคืึทืื ืืื ืื ืงืืื ืืืึธืก ืืืจ ืืึทืจืคึฟื.
ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืฉืืขืื cacheDataBlockPercent = 20 ืืื ืืขื ืืืึธืก ืืึทืคึผืึทื ื:
ืืขืจ ืจืขืืืืืึทื ืืื ืงืืึธืจ ืืื ืืขืจ ืืึธื. ืืื ืื ืืจืึทืคืก ืืื ืื, ืขืก ืืืขืจื ืงืืึธืจ ืืืึธืก ืึทืืึท ืึท ืึทืงืกืขืืขืจืืืฉืึทื ืืื ืคืืจืืขืงืืืขื - ืืืจ ืฉืคึผืึธืจื ืึท ืคึผืืึทืฅ ืคืื GC ืจืขืกืืจืกื ืึธื ืืื ืื ืกืืกืืคืขืึทื ืึทืจืืขื ืคืื ืคึผืืืืกืื ื ืืึทืื ืืื ืื ืงืึทืฉ ืืืืื ืฆื ืืื ืืืึทืจืคื ืขืก ืึทืจืึธืคึผ ืื ืคืืืกื ืคืื ืื ืืึทืจืฉืึทื ืืื ื:
ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืงืคึผื ืืืืึทืืึทืืืืฉืึทื ืื ืงืจืืกืื, ืึธืืขืจ ืืื ืคืื ืืืืื ืืงืขืจ ืืื ืคึผืจืึธืืืึทืงืืืืืืื:
ืขืก ืืื ืืืื ืืืื ืฆื ืืืืขืจืงื ืึทื ืื ืืืึทืงืก ืกืืึธืจื ืืื BlockCache ืืขื ืขื ืึทื ืืขืจืฉ. ืจืืึฟ, ืืืขืื 95%, ืืื ืืึทืื ืืื. ืืื ืื ืื ืืื ืืื ืืขืืึทืืึทืืึท, ืึทืืึท ืืื ืืืื ืคืืืืขืจืก ืึธืืขืจ LEAF_INDEX ืืื
ืืขืจืืืขืจ, ืืื ืื ืงืึธื ืืืจ ืืขื ืึท ืืฉืขืง ืฆืืฉืืึทื ื buf.getBlockType().isData() ืืื ืืึทื ืง ืฆื ืืขื ืืขืืึท, ืืืจ ืืืขืื ืืึธืื ืขืก ืืื ืื ืงืึทืฉ ืืื ืงืืื ืคืึทื.
ืืืฆื ืืึธืื ืก ืคืึทืจืืจืขืกืขืจื ืื ืืึทืกืข ืืื ืึท ืืืกื ืคืึทืจืฉืืืึทืคื ืื ืฉืืจืื ืืื ืืืื ืืึธื. ืืื ืืขืจ ืขืจืฉืืขืจ ืคึผืจืึธืืข ืืืจ ืืขืืืื ืื ืงืึทืืึธืฃ ืคึผืจืึธืฆืขื ื = 20 ืืื BlockCache ืืื ืืขืืืขื ืึท ืืืกื ืึทื ืืขืจืืืืึทืืืืื. ืืืฆื ืืึธืื ืก ืฉืืขืื ืขืก ืฆื 23% ืืื ืืืืื 100 ืคึฟืขืืขื ืืขืืขืจ 5 ืืื ืื ืฆื ืืขื ืืื ืืืึธืก ืคืื ื ืืขืืืงืื ื ืึทืงืขืจื:
ืืึธ ืืืจ ืืขื ืึทื ืืขืจ ืึธืจืืืื ืขื ืืืขืจืกืืข ืึผืืขื ืืืืื ืฉืืึธืื ืื ืกืืคืื ืืื ืืืขืื 100 ืืืืื ื ืจืืงืืืขืก ืคึผืขืจ ืกืขืงืื ืืข. ืืืืขืจืึทื ืื ืืึทืืข ืืื ืึท ืึทืงืกืขืืขืจืืืฉืึทื ืคืื ืึทืจืืืฃ ืฆื 300 ืืืืื ื. ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืขืก ืืื ืงืืึธืจ ืึทื ืืืืึทืืขืจ ืึทืงืกืขืืขืจืืืฉืึทื ืืื ื ืื ืืขืจ ืึทืืื "ืคืจืื"; ืงืคึผื ืืืืึทืืึทืืืืฉืึทื ืืื ืืืื ืื ืงืจืืกืื ื.
ืึธืืขืจ, ืืึธืก ืืื ื ืืฉื ืึท ืืืืขืจ ืขืืขืืึทื ื ืืืืืื ื, ืืืืึทื ืืืจ ืืึธื ื ืื ืืืืกื ืืื ืฉืืืึทืื ืืืึธืก ืคึผืจืึธืฆืขื ื ืคืื ืืืึทืงืก ืืึธื ืืืื ืงืึทืฉื, ืขืก ืืขืคึผืขื ืืก ืืืืฃ ืื ืืึทืกืข ืคึผืจืึธืคืื. ืืขืจืืืขืจ, ืึท ืืขืงืึทื ืืืึทื ืืื ืืืคึผืืึทืืขื ืึทื ืฆื ืืืืืึธืืึทืืืฉ ืกืืจืืืขืจื ืืขื ืคึผืึทืจืึทืืขืืขืจ ืืืคึผืขื ืืื ื ืืืืฃ ืื ืืขืืืงืืื ืคืื ืืืืขื ืขื ืึทืคึผืขืจืืืฉืึทื ื.
ืืจืื ืึธืคึผืฆืืขืก ืืขื ืขื ืฆืืืขืืขืื ืฆื ืงืึธื ืืจืึธืืืจื ืืขื:
hbase.lru.cache.heavy.eviction.count.limit - ืฉืืขืื ืืื ืคืืืข ืืึธื ืืขืจ ืคึผืจืึธืฆืขืก ืคืื ืืืืืงืืื ื ืืึทืื ืคืื ืื ืงืึทืฉ ืืึธื ืืืืคื ืืืืืขืจ ืืืจ ืึธื ืืืืื ื ืืฆื ืึทืคึผืืึทืืึทืืืืฉืึทื (ื"ื ืกืงืืคึผืื ื ืืืึทืงืก). ืืืจื ืคืขืืืงืืึทื ืขืก ืืื ืืืืึทื ืฆื MAX_INT = 2147483647 ืืื ืืื ืคืึทืงื ืึทื ืื ืฉืืจืื ืืืขื ืงืืื ืืึธื ืึธื ืืืืื ืืจืืขืื ืืื ืืขื ืืืขืจื. ืืืืึทื ืื ืืืืืงืฉืึทื ืคึผืจืึธืฆืขืก ืกืืึทืจืฅ ืืขืืขืจ 5 - 10 ืกืขืงืื ืืขืก (ืขืก ืืขืคึผืขื ืืก ืืืืฃ ืื ืืึทืกืข) ืืื 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 ืืืจ. ืึธืืขืจ, ืืืจ ืงืขื ืขื ืฉืืขืื ืืขื ืคึผืึทืจืึทืืขืืขืจ ืฆื 0 ืืื ืืึทืื ืื ืฉืืจืื ืึทืจืืขื ืืืืื ื ืึธื ืงืึทืืขืจ.
ืึธืืขืจ, ืขืก ืืื ืืืื ืึท ืคึผืืืืึธืื ืืื ืืขื ืคึผืึทืจืึทืืขืืขืจ. ืืืื ืืื ืืืขืจ ืืึทืกืข ืืื ืึทืืื ืึทื ืงืืจืฅ-ืืขืจืืื ืืืืขื ืขื (ืืึธืื ืืขืฉืึทืก ืืขื ืืึธื) ืืื ืืึทื ื-ืืขืจืืื ืืืืขื ืขื (ืืื ื ืึทืื) ืืขื ืขื ืงืขืกืืืืขืจ ืื ืืขืจืกืคึผืขืจืกื, ืืืจ ืงืขื ืขื ืืึทืื ืืืืขืจ ืึทื ืื ืฉืืจืื ืืื ืืืืกืืขืืจืืื ืืืืฃ ืืืืื ืืืขื ืืึทื ื ืืืืขื ืขื ืึทืคึผืขืจืืืฉืึทื ื ืืขื ืขื ืืื ืคึผืจืึธืืจืขืก.
ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืืจ ืืืืกื ืึทื ืงืืจืฅ-ืืขืจืืื ืจืืืื ืื ืืืืฉืึทืืืึทืื ืืขืฆืืข ืืืขืื 1 ืืื ืื. ืขืก ืืื ื ืื ืืึทืจืคึฟื ืฆื ืึธื ืืืืื ืคืืจืืืืจืคื ืืืึทืงืก, ืืขืจ ืงืึทืฉ ืืืขื ื ืืฉื ืืึธืื ืฆืืึทื ืฆื ืืืขืจื ืึทืืืืืืืื ืืื ืืขืืึธืื ืืืจ ืงืขื ืขื ืฉืืขืื ืืขื ืคึผืึทืจืึทืืขืืขืจ ืืืืึทื ืฆื, ืืืฉื, 10. ืืึธืก ืืืขื ืคืืจื ืฆื ืื ืคืึทืงื ืึทื ืื ืึทืคึผืืึทืืึทืืืืฉืึทื ืืืขื ืึธื ืืืืื ืืจืืขืื ืืืืื ืืืขื ืืึทื ื- ืืขืจืืื ืึทืงืืืื ืืืืขื ืขื ืืื ืื ืืขืืืืื, ื.ื. ืืื 100 ืกืขืงืื ืืขืก. ืืืื, ืืืื ืืืจ ืืึธืื ืึท ืงืืจืฅ-ืืขืจืืื ืืืืขื ืขื, ืึทืืข ืืืึทืงืก ืืืขื ืืืื ืืื ืื ืงืึทืฉ ืืื ืืืขื ืืืื ืืืจืขืืืืื (ืึทืืืฅ ืคึฟืึทืจ ืื ืืืึธืก ืืืขื ืืืื ืืืืืงืืื ืืืจื ืื ื ืึธืจืืึทื ืึทืืืขืจืืืึทื). ืืื ืืืขื ืืืจ ืืึธื ืืึทื ื-ืืขืจืืื ืืืืขื ืขื, ืื ืฉืืจืื ืืื ืคืืจืงืขืจื ืืืืฃ ืืื ืืืจ ืืืึธืื ืืึธืื ืคืื ืืขืืขืจ ืคืึธืจืฉืืขืืื ื.
hbase.lru.cache.heavy.eviction.mb.size.limit - ืฉืืขืื ืืื ืคืืืข ืืขืืืืืื ืืืจ ืืืึธืื ืืื ืฆื ืฉืืขืื ืืื ืื ืงืึทืฉ (ืืื, ืคืื ืงืืจืก, ืืืืืงื) ืืื 10 ืกืขืงืื ืืขืก. ืืขืจ ืฉืืจืื ืืืขื ืคึผืจืืืืจื ืฆื ืืขืจืืจืืืื ืืขื ืืืขืจื ืืื ืืึทืืื ืขืก. ืื ื ืงืืื ืืื ืืืก: ืืืื ืืืจ ืฉืืืคึผื ืืืืืืืื ืืจืืื ืืื ืขื ืงืืฉ, ืืืจืคื ืืืจ ืืจืืืกืืืืจืคื ืืืืืืืื, ืืื ืืืก, ืืื ืืืจ ืืืื ืืืืื ืืขืืขื, ืืื ืืืืขืจ ืืืืขืจ. ืึธืืขืจ, ืืืจ ืืึธื ื ืืฉื ืคึผืจืืืืจื ืฆื ืฉืืขืื ืขืก ืฆื ืงืืืื, ืืืืึทื ืืึธืก ืืืขื ืคืึทืจืฉืึทืคื ืคึผืจืืืึทืืฉืืจืื ืึทืจืืืกืืึทื ื ืื ืืืึธืง ืืึธืคึผืงืขื ืืึธืืข. ืคึฟืึทืจ ืฉืืึทืจืง ืกืขืจืืืขืจืก (ืืืขืื 20-40 ืืฉืืืืช ืงืึธืจืขืก), ืขืก ืืื ืึธืคึผืืืืึทื ืฆื ืฉืืขืื ืืืขืื 300-400 ืืขืืืืืืื. ืคึฟืึทืจ ืื ืืืื ืงืืึทืก (~ 10 ืงืึธืจืขืก) 200-300 ืืขืืืืืืื. ืคึฟืึทืจ ืฉืืืึทื ืกืืกืืขืืขื (2-5 ืงืึธืจืขืก) 50-100 ืืขืืืืืืื ืงืขื ืืืื ื ืึธืจืืึทื (ื ืื ืืขืกืืขื ืืืืฃ ืื).
ืืึธืืืจ ืงืืงื ืืื ืืึธืก ืึทืจืืขื: ืืึธืืืจ ืืึธืื ืึทื ืืืจ ืฉืืขืื hbase.lru.cache.heavy.eviction.mb.size.limit = 500, ืขืก ืืื ืึท ืกืึธืจื ืคืื ืืึทืกืข (ืืืืขื ืขื) ืืื ืืขืืขืจ ~ 10 ืกืขืงืื ืืขืก ืืืจ ืจืขืืขื ืขื ืืื ืคืืืข ืืืืขืก ืืขื ืขื ืืขืืืขื ืขืืืืงืืื ืืื ืื ืคืึธืจืืืืข:
ืึธืืืืขืจืืขื = ืคืจืื ืืืืขืก ืกืึทื (ืื) * 100 / ืืืืื (ืื) - 100;
ืืืื ืืื ืคืึทืงื 2000 ืืขืืืืืืื ืืขื ืขื ืืืืืงืืื, ืื ืึธืืืืขืจืืขื ืืื ืืืืึทื ืฆื:
2000 * 100 / 500 - 100 = 300%
ืื ืึทืืืขืจืืืึทืื ืคึผืจืืืืจื ืฆื ืืึทืืื ื ืื ืืขืจ ืืื ืึท ืืืกื ืืขื ืก ืคืื ืคึผืจืึธืฆืขื ื, ืึทืืื ืื ืฉืืจืื ืืืขื ืจืขืืืฆืืจื ืื ืคึผืจืึธืฆืขื ื ืคืื ืงืึทืฉื ืืืึทืงืก, ืืื ืืขืจืืื ืืืคึผืืึทืืขื ืื ื ืึทื ืึทืืืึธ-ืืื ืื ื ืืขืงืึทื ืืืึทื.
ืึธืืขืจ, ืืืื ืื ืืึทืกืข ืืจืืคื ืก, ืืึธืื ืืื ืื ืืึธืื ืืืืื 200 ืืขืืืืืืื ืืขื ืขื ืืืืืงืืื ืืื ืึธืืืืขืจืืขื ืืืขืจื ื ืขืืึทืืืื (ืื ืึทืืื ืืขืจืืคืขื ืข ืึธืืืืขืจืฉืึธืึธืืื ื):
200 * 100 / 500 - 100 = -60%
ืืืืฃ ืื ืคืึทืจืงืขืจื, ืื ืฉืืจืื ืืืขื ืคืึทืจืืจืขืกืขืจื ืื ืคึผืจืึธืฆืขื ื ืคืื ืงืึทืฉื ืืืึทืงืก ืืื ืึธืืืขืจืืขืึทื ืืืขืจื positive.
ืื ืืขืจ ืืื ืึท ืืืืฉืคึผืื ืคืื ืืื ืืึธืก ืงืืงื ืืืืฃ ืคืึทืงืืืฉ ืืึทืื. ืขืก ืืื ื ืื ืืึทืจืคึฟื ืฆื ืคึผืจืืืืจื ืฆื ืืขืจืืจืืืื 0%, ืขืก ืืื ืืืืืขืืืขื. ืขืก ืืื ืืืืขืจ ืืื ืืืขื ืขืก ืืื ืืืขืื 30-100%, ืืึธืก ืืขืืคึผืก ืฆื ืืืกืืืืื ืฆื ืคืจื ืึทืจืืืกืืึทื ื ืคืื ืื ืึทืคึผืืึทืืึทืืืืฉืึทื ืืึธืืข ืืขืฉืึทืก ืงืืจืฅ-ืืขืจืืื ืกืขืจืืืฉื.
hbase.lru.cache.heavy.eviction.overhead.coefficient - ืฉืืขืื ืืื ืืขืฉืืืื ื ืืืจ ืืืึธืื ืืื ืฆื ืืึทืงืืืขื ืื ืจืขืืืืืึทื. ืืืื ืืืจ ืืืืกื ืคึฟืึทืจ ืืืืขืจ ืึทื ืืื ืืืขืจ ืืืืขื ืขื ืืขื ืขื ืืขืจืกืื ืก ืืึทื ื ืืื ืืึธื ื ืืฉื ืืืขืื ืฆื ืืืึทืจืื, ืืืจ ืงืขื ืขื ืคืึทืจืืจืขืกืขืจื ืืขื ืคืึทืจืืขืืืขื ืืฉ ืืื ืืึทืงืืืขื ืืืื ืคืึธืจืฉืืขืืื ื ืคืึทืกืืขืจ.
ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืืจ ืฉืืขืื ืืขื ืงืึธืืึทืคืืฉืึทื ื = 0.01. ืืขื ืืืื ืึทื ืึธืืืขืจืืขืึทื (ืืขื ืืืืื) ืืืขื ืืืื ืืขืืขืจื ืืื ืืขื ื ืืืขืจ ืืืจื ืื ืจืืืึทืืืื ื ืจืขืืืืืึทื ืืื ืืขืจ ืคึผืจืึธืฆืขื ื ืคืื ืงืึทืฉื ืืืึทืงืก ืืืขื ืืืื ืจืืืืกื. ืืึธืืืจ ืืืขืจื ืขืืขื ืึทื ืึธืืืขืจืืขืึทื = 300% ืืื ืงืึธืืึทืคืืฉืึทื ื = 0.01, ืืขืืึธืื ืืขืจ ืคึผืจืึธืฆืขื ื ืคืื ืงืึทืฉื ืืืึทืงืก ืืืขื ืืืื ืจืืืืกื ืืื 3%.
ื ืขื ืืขื "ืืึทืงืงืคึผืจืขืกืกืืจืข" ืืึธืืืง ืืื ืืืื ืืืคึผืืึทืืขื ืึทื ืคึฟืึทืจ ื ืขืืึทืืืื ืึธืืืขืจืืขืึทื (ืึธืืืขืจืฉืึธืึธืืื ื) ืืืึทืืืขืก. ืืื ื ืงืืจืฅ-ืืขืจืืื ืคืืึทืงืืฉืืืืฉืึทื ื ืืื ืื ืืึทื ื ืคืื ืืืืขื ืขื ืืื ืืืืืงืฉืึทื ื ืืขื ืขื ืฉืืขื ืืืง ืืขืืืขื, ืืขื ืืขืงืึทื ืืืึทื ืึทืืึทืื ืืืจ ืฆื ืืืกืืืืื ืฆื ืคืจื ืึทืจืืืกืืึทื ื ืคืื ืื ืึทืคึผืืึทืืึทืืืืฉืึทื ืืึธืืข. ืืึทืงืงืคึผืจืขืกืกืืจืข ืืื ืึท ืื ืืืขืจืืื ืืึธืืืง: ืื ืฉืืืจืงืขืจ ืื ืึธืืืืขืจืฉืึธืึธืืื ื, ืื ืืขืจ ืืืึทืงืก ืืขื ืขื ืงืึทืฉื.
ืืืคึผืืขืืขื ืืึทืืืึธื ืงืึธื
LruBlockCache cache = this.cache.get();
if (cache == null) {
break;
}
freedSumMb += cache.evict()/1024/1024;
/*
* Sometimes we are reading more data than can fit into BlockCache
* and it is the cause a high rate of evictions.
* This in turn leads to heavy Garbage Collector works.
* So a lot of blocks put into BlockCache but never read,
* but spending a lot of CPU resources.
* Here we will analyze how many bytes were freed and decide
* decide whether the time has come to reduce amount of caching blocks.
* It help avoid put too many blocks into BlockCache
* when evict() works very active and save CPU for other jobs.
* More delails: https://issues.apache.org/jira/browse/HBASE-23887
*/
// First of all we have to control how much time
// has passed since previuos evict() was launched
// This is should be almost the same time (+/- 10s)
// because we get comparable volumes of freed bytes each time.
// 10s because this is default period to run evict() (see above this.wait)
long stopTime = System.currentTimeMillis();
if ((stopTime - startTime) > 1000 * 10 - 1) {
// Here we have to calc what situation we have got.
// We have the limit "hbase.lru.cache.heavy.eviction.bytes.size.limit"
// and can calculte overhead on it.
// We will use this information to decide,
// how to change percent of caching blocks.
freedDataOverheadPercent =
(int) (freedSumMb * 100 / cache.heavyEvictionMbSizeLimit) - 100;
if (freedSumMb > cache.heavyEvictionMbSizeLimit) {
// Now we are in the situation when we are above the limit
// But maybe we are going to ignore it because it will end quite soon
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
// It is going for a long time and we have to reduce of caching
// blocks now. So we calculate here how many blocks we want to skip.
// It depends on:
// 1. Overhead - if overhead is big we could more aggressive
// reducing amount of caching blocks.
// 2. How fast we want to get the result. If we know that our
// heavy reading for a long time, we don't want to wait and can
// increase the coefficient and get good performance quite soon.
// But if we don't sure we can do it slowly and it could prevent
// premature exit from this mode. So, when the coefficient is
// higher we can get better performance when heavy reading is stable.
// But when reading is changing we can adjust to it and set
// the coefficient to lower value.
int change =
(int) (freedDataOverheadPercent * cache.heavyEvictionOverheadCoefficient);
// But practice shows that 15% of reducing is quite enough.
// We are not greedy (it could lead to premature exit).
change = Math.min(15, change);
change = Math.max(0, change); // I think it will never happen but check for sure
// So this is the key point, here we are reducing % of caching blocks
cache.cacheDataBlockPercent -= change;
// If we go down too deep we have to stop here, 1% any way should be.
cache.cacheDataBlockPercent = Math.max(1, cache.cacheDataBlockPercent);
}
} else {
// Well, we have got overshooting.
// Mayby it is just short-term fluctuation and we can stay in this mode.
// It help avoid permature exit during short-term fluctuation.
// If overshooting less than 90%, we will try to increase the percent of
// caching blocks and hope it is enough.
if (freedSumMb >= cache.heavyEvictionMbSizeLimit * 0.1) {
// Simple logic: more overshooting - more caching blocks (backpressure)
int change = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += change;
// But it can't be more then 100%, so check it.
cache.cacheDataBlockPercent = Math.min(100, cache.cacheDataBlockPercent);
} else {
// Looks like heavy reading is over.
// Just exit form this mode.
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
freedSumMb, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
freedSumMb = 0;
startTime = stopTime;
}
ืืื ืก ืืืฆื ืงืืง ืืืึท ืึทืืข ืืขื ื ืืฆื ืึท ืคืึทืงืืืฉ ืืืึทืฉืคึผืื. ืืืจ ืืึธืื ืื ืคืืืืขื ืืข ืคึผืจืึธืืข ืฉืจืืคื:
- ืืึธืืืจ ืึธื ืืืืื ืกืงืึทื (25 ืคึฟืขืืขื, ืคึผืขืงื = 100)
- ื ืึธื 5 ืืื ืื, ืืืืื ืืึทืืื-ืืขืฅ (25 ืคึฟืขืืขื, ืคึผืขืงื = 100)
- ื ืึธื 5 ืืื ืื, ืงืขืจ ืึทืืืขืง ืืืืื-ืืขืฅ (ืืืืื ืืืขืจืงืืงื ืืืืืื ืืืืืขืจ)
ืืืจ ืืึธื ืฆืืืื ืจืึทื ื, ืขืจืฉืืขืจ hbase.lru.cache.heavy.eviction.count.limit = 10000 (ืืืึธืก ืึทืงืืฉืึทืืืึทืื ืืืกืืืืึทืื ืื ืฉืืจืื), ืืื ืืขืืึธืื ืฉืืขืื ืืืืื = 0 (ืขื ืืืืึทืื ืขืก).
ืืื ืื ืืึธืืก ืืื ืื ืืืจ ืืขื ืืื ืื ืฉืืจืื ืืื ืืืืกืืขืืจืืื ืืืืฃ ืืื ืจืืกืขื ืึธืืืขืจืฉืึธืึธืืื ื ืฆื 14-71%. ืคืื ืฆืืื ืฆื ืฆืืื ืื ืืึทืกืข ืืืงืจืืกืึทื, ืืืึธืก ืืืจื ืก ืืืืฃ ืืึทืงืงืคึผืจืขืกืกืืจืข ืืื HBase ืงืึทืฉ ืืขืจ ืืืึทืงืก ืืืืืขืจ.
ืงืืึธืฅ ืจืขืืืึธื ืกืขืจืืืขืจ
ืืืืืงืืื (ืื): 0, ืคืึทืจืืขืืืขื ืืฉ 0.0, ืึธืืืืขืจืืขื (%): -100, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 0, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 100
ืืืืืงืืื (ืื): 0, ืคืึทืจืืขืืืขื ืืฉ 0.0, ืึธืืืืขืจืืขื (%): -100, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 0, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 100
ืืืืืงืืื (ืื): 2170, ืคืึทืจืืขืืืขื ืืฉ 1.09, ืึธืืืืขืจืืขื (%): 985, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 1, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 91 < ืึธื ืืืื
ืืืืืงืืื (ืื): 3763, ืคืึทืจืืขืืืขื ืืฉ 1.08, ืึธืืืืขืจืืขื (%): 1781, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 2, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 76
ืืืืืงืืื (ืื): 3306, ืคืึทืจืืขืืืขื ืืฉ 1.07, ืึธืืืืขืจืืขื (%): 1553, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 3, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 61
ืืืืืงืืื (ืื): 2508, ืคืึทืจืืขืืืขื ืืฉ 1.06, ืึธืืืืขืจืืขื (%): 1154, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 4, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 50
ืืืืืงืืื (ืื): 1824, ืคืึทืจืืขืืืขื ืืฉ 1.04, ืึธืืืืขืจืืขื (%): 812, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 5, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 42
ืืืืืงืืื (ืื): 1482, ืคืึทืจืืขืืืขื ืืฉ 1.03, ืึธืืืืขืจืืขื (%): 641, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 6, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 36
ืืืืืงืืื (ืื): 1140, ืคืึทืจืืขืืืขื ืืฉ 1.01, ืึธืืืืขืจืืขื (%): 470, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 7, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 32
ืืืืืงืืื (ืื): 913, ืคืึทืจืืขืืืขื ืืฉ 1.0, ืึธืืืืขืจืืขื (%): 356, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 8, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 29
ืืืืืงืืื (ืื): 912, ืคืึทืจืืขืืืขื ืืฉ 0.89, ืึธืืืืขืจืืขื (%): 356, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 9, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 26
ืืืืืงืืื (ืื): 684, ืคืึทืจืืขืืืขื ืืฉ 0.76, ืึธืืืืขืจืืขื (%): 242, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 10, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 24
ืืืืืงืืื (ืื): 684, ืคืึทืจืืขืืืขื ืืฉ 0.61, ืึธืืืืขืจืืขื (%): 242, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 11, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.51, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 12, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 21
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.42, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 13, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 20
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 14, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 15, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.32, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 16, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.31, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 17, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.3, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 18, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.29, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 19, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.27, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 20, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.25, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 21, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.24, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 22, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.22, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 23, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.21, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 24, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.2, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 25, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.17, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 26, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.17, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 27, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18 < ืฆืืืขืืขืื ืืขืฅ (ืึธืืขืจ ืื ืืขืืืข ืืืฉ)
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.15, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 28, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.13, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 29, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.11, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 30, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.09, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 31, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.08, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 32, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.07, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 33, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.06, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 34, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.05, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 35, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.05, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 36, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.04, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 37, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 109, ืคืึทืจืืขืืืขื ืืฉ 0.04, ืึธืืืืขืจืืขื (%): -46, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 37, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึธืงืง (%): 22 < ืฆืืจืืง ืืจืืง
ืืืืืงืืื (ืื): 798, ืคืึทืจืืขืืืขื ืืฉ 0.24, ืึธืืืืขืจืืขื (%): 299, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 38, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 20
ืืืืืงืืื (ืื): 798, ืคืึทืจืืขืืืขื ืืฉ 0.29, ืึธืืืืขืจืืขื (%): 299, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 39, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 570, ืคืึทืจืืขืืืขื ืืฉ 0.27, ืึธืืืืขืจืืขื (%): 185, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 40, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 17
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.22, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 41, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.16, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 42, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.11, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 43, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.09, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 44, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.07, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 45, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.05, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 46, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (ืื): 222, ืคืึทืจืืขืืืขื ืืฉ 0.04, ืึธืืืืขืจืืขื (%): 11, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 47, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 16
ืืืืืงืืื (MB): 104, ืคืึทืจืืขืืืขื ืืฉ 0.03, ืึธืืืืขืจืืขื (%): -48, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 47, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึธืงืง (%): 21 < ืืืขืจืจืืึทืก ืืขืฅ
ืืืืืงืืื (ืื): 684, ืคืึทืจืืขืืืขื ืืฉ 0.2, ืึธืืืืขืจืืขื (%): 242, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 48, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 19
ืืืืืงืืื (ืื): 570, ืคืึทืจืืขืืืขื ืืฉ 0.23, ืึธืืืืขืจืืขื (%): 185, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 49, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.22, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 50, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.21, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 51, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.2, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 52, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.18, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 53, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.16, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 54, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 228, ืคืึทืจืืขืืืขื ืืฉ 0.14, ืึธืืืืขืจืืขื (%): 14, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 55, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 18
ืืืืืงืืื (ืื): 112, ืคืึทืจืืขืืืขื ืืฉ 0.14, ืึธืืืืขืจืืขื (%): -44, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 55, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึธืงืง (%): 23 < ืฆืืจืืง ืืจืืง
ืืืืืงืืื (ืื): 456, ืคืึทืจืืขืืืขื ืืฉ 0.26, ืึธืืืืขืจืืขื (%): 128, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 56, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.31, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 57, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 58, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 59, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 60, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 61, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 62, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 63, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.32, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 64, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 65, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 66, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.32, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 67, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 68, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.32, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 69, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.32, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 70, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 71, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 72, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 73, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 74, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 75, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 342, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): 71, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 76, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 22
ืืืืืงืืื (ืื): 21, ืคืึทืจืืขืืืขื ืืฉ 0.33, ืึธืืืืขืจืืขื (%): -90, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 76, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 32
ืืืืืงืืื (ืื): 0, ืคืึทืจืืขืืืขื ืืฉ 0.0, ืึธืืืืขืจืืขื (%): -100, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 0, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 100
ืืืืืงืืื (ืื): 0, ืคืึทืจืืขืืืขื ืืฉ 0.0, ืึธืืืืขืจืืขื (%): -100, ืฉืืืขืจ ืืืืืงืฉืึทื ืืึธืืืึทื ืง: 0, ืงืจืึทื ื ืงืึทืืฉืื ื ืืึทืืึทืืืึทืง (%): 100
ืื ืกืงืึทื ื ืืขื ืขื ืืืจืฃ ืฆื ืืืืึทืื ืื ืืขืืืข ืคึผืจืึธืฆืขืก ืืื ืื ืคืึธืจืขื ืคืื ืึท ืืจืึทืคืืง ืคืื ืื ืฉืืืืืช ืฆืืืืฉื ืฆืืืื ืงืึทืฉ ืกืขืงืฉืึทื ื - ืืืื (ืืื ืืืึทืงืก ืืืึธืก ืืึธืื ืงืืื ืืึธื ืืขืืืขื ืืขืืขืื ืคืจืืขืจ) ืืื ืืืืื (ืืึทืืึท "ืืขืืขืื" ืืื ืืื ืืกืืขืจ ืึทืืึธื ืืขื ืขื ืกืืึธืจื ืืึธ):
ืืื ืืขืกืึธืฃ, ืืื ืื ืึธืคึผืขืจืึทืฆืืข ืคืื โโืื ืคึผืึทืจืึทืืขืืขืจืก ืงืืง ืืื ืืื ืื ืคืึธืจืขื ืคืื ืึท ืืจืึทืคืืง. ืคึฟืึทืจ ืคืึทืจืืืืึทื, ืื ืงืึทืฉ ืืื ืืึธืจ ืืืืกืืขืืจืืื ืืื ืื ืึธื ืืืื, ืืื HBase ืืื ืืึธื ืืฉื ืืื ืงืึทืืฉืื ื ืืื ืคืึทืจืืึทืืื ืื ืึธื ืืืื ืคืื ืึทืคึผืืึทืืึทืืืืฉืึทื ืึทืจืืขื ืืื 5 ืืื ืื (30 ืืืืืงืฉืึทื ืกืืืงืึทืื).
ืืึทื ืฅ ืงืึธื ืงืขื ืขื ืืืื ืืขืคึฟืื ืขื ืืื Pull Request
ืึธืืขืจ, 300 ืืืืื ื ืืืืขื ืขื ืคึผืขืจ ืกืขืงืื ืืข ืืื ื ืื ืึทืืข ืืืึธืก ืงืขื ืขื ืืืื ืึทืืฉืืืื ืืืืฃ ืืขื ืืึทืื ืืืึทืจื ืืื ืืขืจ ืื ืืืืื ืืื ืืขื. ืืขืจ ืคืึทืงื ืืื ืึทื ืืืขื ืืืจ ืืึทืจืคึฟื ืฆื ืึทืงืกืขืก ืืึทืื ืืืจื HDFS, ืื ShortCircuitCache (ืืขืจื ืึธื ืจืืคืขืจื ืฆื ืืื SSC) ืืขืงืึทื ืืืึทื ืืื ืืขื ืืฆื, ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืึทืงืกืขืก ืื ืืึทืื ืืืืึทื, ืืืกืืืืื ื ืขืฅ ืื ืืขืจืึทืงืฉืึทื ื.
ืคึผืจืึธืคืืืื ื ืืึธื ืืขืืืืื ืึทื ืืึธืืฉ ืืขืจ ืืขืงืึทื ืืืึทื ืืื ืึท ืืจืืืก ืืขืืืื ืก, ืขืก ืืืื ืืื ืขืืืขืืข ืคืื ื ืืืขืจื ืึท ืืึทืืึทืื ืขืง, ืืืืึทื ืึผืืขื ืึทืืข ืฉืืืขืจ ืึทืคึผืขืจืืืฉืึทื ื ืคืึทืื ืืื ืึท ืฉืืึธืก, ืืืึธืก ืคืืจื ืฆื ืืืึทืงืื ื ืจืืึฟ ืคืื ืื ืฆืืื.
ืืืขื ืืืจ ืืึธืื ืืืื ืืขืืขื ืืขื, ืืืจ ืืืื ืืขืืขื ืึทื ืื ืคึผืจืึธืืืขื ืงืขื ืขื ืืืื ืกืขืจืงืึทืืืืขื ืืื ืืืจื ืงืจืืืืืื ื ืึท ืืขื ืืข ืคืื โโโโืคืจืืึท SSCs:
private final ShortCircuitCache[] shortCircuitCache;
...
shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
for (int i = 0; i < this.clientShortCircuitNum; i++)
this.shortCircuitCache[i] = new ShortCircuitCache(โฆ);
ืืื ืืขืืึธืื ืึทืจืืขื ืืื ืืื, ืขืงืกืงืืืืื ื ืื ืืขืจืกืขืงืฉืึทื ื ืืืื ืืืึท ืื ืืขืฆืืข ืคืึธืืึธ ืฆืืคึฟืขืจ:
public ShortCircuitCache getShortCircuitCache(long idx) {
return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
}
ืืืฆื ืืืจ ืงืขื ืขื ืึธื ืืืืื ืืขืกืืื ื. ืฆื ืืึธื ืืึธืก, ืืืจ ืืืขืื ืืืืขื ืขื ืืขืงืขืก ืคึฟืื HDFS ืืื ืึท ืคึผืฉืื ืืืืื-ืืจืขืืื ืึทืคึผืืึทืงืืืฉืึทื. ืฉืืขืื ืื ืคึผืึทืจืึทืืขืืขืจืก:
conf.set("dfs.client.read.shortcircuit", "true");
conf.set("dfs.client.read.shortcircuit.buffer.size", "65536"); // ะฟะพ ะดะตัะพะปัั = 1 ะะ ะธ ััะพ ัะธะปัะฝะพ ะทะฐะผะตะดะปัะตั ััะตะฝะธะต, ะฟะพััะพะผั ะปัััะต ะฟัะธะฒะตััะธ ะฒ ัะพะพัะฒะตัััะฒะธะต ะบ ัะตะฐะปัะฝัะผ ะฝัะถะดะฐะผ
conf.set("dfs.client.short.circuit.num", num); // ะพั 1 ะดะพ 10
ืืื ื ืึธืจ ืืืืขื ืขื ืื ืืขืงืขืก:
FSDataInputStream in = fileSystem.open(path);
for (int i = 0; i < count; i++) {
position += 65536;
if (position > 900000000)
position = 0L;
int res = in.read(position, byteBuffer, 0, 65536);
}
ืืขืจ ืงืึธื ืืื ืขืงืกืึทืงืืืืึทื ืืื ืืึทืืื ืืขืจ ืคึฟืขืืขื ืืื ืืืจ ืืืขืื ืคืึทืจืืจืขืกืขืจื ืื ื ืืืขืจ ืคืื ืกืืืืึทืืืืื ืืึทืกืื ืืืืขื ืขื ืืขืงืขืก (ืคืื 10 ืฆื 200 - ืืึธืจืืืึธื ืืึทื ืึทืงืก) ืืื ืื ื ืืืขืจ ืคืื ืงืึทืืฉืขืก (ืคืื 1 ืฆื 10 - ืืจืึทืคืืงืก). ืื ืืืขืจืืืงืึทื ืึทืงืก ืืืืืื ืื ืึทืงืกืขืืขืจืืืฉืึทื ืืืึธืก ืจืืืึทืืืื ืคืื ืึท ืคืึทืจืืจืขืกืขืจื ืืื SSC ืงืึธืจืขืื ืฆื ืื ืคืึทื ืืืขื ืขืก ืืื ืืืืื ืืืื ืงืึทืฉ.
ืืื ืฆื ืืืืขื ืขื ืื ืืจืึทืคืืง: ืื ืืืจืืคืืจืื ื ืฆืืื ืคึฟืึทืจ 100 ืืืืื ื ืืืืขื ืขื ืืื 64 ืงื ืืืึทืงืก ืืื ืืืื ืงืึทืฉ ืจืืงืืืืืขืจื 78 ืกืขืงืื ืืขืก. ืืื 5 ืงืึทืืฉืขืก ืขืก ื ืขืื 16 ืกืขืงืื ืืขืก. ืืขื ืข. ืขืก ืืื ืึท ืึทืงืกืขืืขืจืืืฉืึทื ืคืื ~ 5 ืืื. ืืื ืืขื ืืขืื ืคืื ืื ืืจืืค ืืื ืื ืืืืจืงืื ื ื ืืฉื ืืขืืจ ืืืืขืจืงื ืืื ื ืงืืืื ืข ืฆืื ืคืืจืืืขื ืืืื, ืขืก ืืืืื ืื ืฆื ืฉืคืืื ื ืืืืขืจืงืืืืข ืจืืืข ืืืขื ืขืก ืืขื ืขื ืคืืจืื ืืขืจ ืืื 50 ืคืึธืืขื ืืืืขื ื, ืขืก ืืื ืืืื ืืืืขืจืงืืื ืื ืืขื ืืื ืคืืจืืจืขืกืขืจื ืื ืฆืื SSC ืคืื 6 ืืื ืืืืื ืืื ืึท ืืืืืืืืง ืงืืขื ืขืจืขืจ ืคืึธืจืฉืืขืืื ื ืคืึทืจืืจืขืกืขืจื.
ืืึทืืขืจืงืื ื 1: ืืื ื ืื ืคึผืจืึธืืข ืจืขืืืืืึทืื ืืขื ืขื ืืึทื ืฅ ืืืึทืืึทืืึทื (ืืขื ืืืืืืขืจ), 3 ืจืึทื ื ืืขื ืขื ืืืจืืืขืงืึธืื ืืื ืื ืจืืืึทืืืื ื ืืืึทืืืขืก ืืขื ืขื ืึทืืืจืืืืฉื.
ืืึทืืขืจืงืื ื 2: ืื ืคืึธืจืฉืืขืืื ื ืืขืืืื ืขื ืคืื ืงืึทื ืคืืืืขืจืื ื ืืจืึทืค ืึทืงืกืขืก ืืื ืื ืืขืืืข, ืืึธืืฉ ืื ืึทืงืกืขืก ืืื ืืื ืึท ืืืกื ืกืืึธืืขืจ.
ืึธืืขืจ, ืขืก ืืื ื ืืืืืง ืฆื ืืขืจืงืืขืจื ืึทื, ื ืื ืขื ืืขื ืืขื ืคืึทื ืืื HBase, ืื ืึทืงืกืขืืขืจืืืฉืึทื ืืื ื ืื ืฉืืขื ืืืง ืคืจืื. ืืึธ ืืืจ "ืืคืฉืืืกื" ืื ืงืคึผื ืก ืคืืืืงืืื ืฆื ืึทืจืืขืื ืืขืจ, ืึทื ืฉืืึธื ืคืื ืืึทื ืืืื ื ืืืืฃ ืืึทืงืก.
ืืึธ ืืืจ ืงืขื ืขื ืึธืืกืขืจืืืืจื ืึทื, ืืื ืึทืืืขืืืื, ืึท ืคืึทืจืืจืขืกืขืจื ืืื ืื ื ืืืขืจ ืคืื ืงืึทืืฉืขืก ืืื ืึท ืืขืขืจืขื ืคึผืจืึทืคึผืึธืจืฉืึทื ืึทื ืคืึทืจืืจืขืกืขืจื ืืื ืงืคึผื ืืืืึทืืึทืืืืฉืึทื. ืึธืืขืจ, ืขืก ืืขื ืขื ืึท ืืืกื ืืขืจ ืืืื ืื ื ืงืึทืืืึทื ืืืฉืึทื ื.
ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืึธืื ืก ื ืขืืขื ืึท ื ืขืขื ืืขืจ ืงืืง ืืืึท ืื ืืึทืฉืืขืืืงื SSC = 3. ืื ืคืึทืจืืจืขืกืขืจื ืืื ืคืึธืจืฉืืขืืื ื ืืืืฃ ืื ืงืืื ืืื ืืืขืื 3.3 ืืื. ืื ืืขืจ ืืขื ืขื ืื ืจืขืืืืืึทืื ืคืื ืึทืืข ืืจืื ืืึทืืื ืืขืจ ืจืึทื ื.
ืืฉืขืช ืงืคึผื ืงืึทื ืกืึทืืฉืึทื ืื ืงืจืืกืื ืืื ืืืขืื 2.8 ืืื. ืืขืจ ืืืืืง ืืื ื ืืฉื ืืืืขืจ ืืจืืืก, ืึธืืขืจ ืื ืงืืืื ืืจืขืืึท ืืื ืฉืืื ืฆืืคืจืืื ืืื ืงืขื ืืึธืื ืฆืืื ืฆื ืืืื ืืื ืฉืืืข ืืื ื ืขืืขื ืืขืงืฆืืขืก.
ืืืื, ืืึธืก ืืืขื ืืึธืื ืึท positive ืืืืจืงืื ื ืคึฟืึทืจ ืงืืื ืืขืฆืืึทื ืืืึธืก ื ืืฆื ืคืึทืจื ืขื ืึทืงืกืขืก ืฆื HDFS (ืืืฉื ืกืคึผืึทืจืง, ืืื"ื ื), ืฆืืืขืฉืืขืื ืึทื ืื ืึทืคึผืืึทืงืืืฉืึทื ืงืึธื ืืื ืืืืืืืืื (ื"ื ืื ืฆืึทืคึผื ืืื ืืืืฃ ืื HDFS ืงืืืขื ื ืืืึทื) ืืื ืขืก ืืื ืคืจืื ืงืคึผื ืืึทืื. . ืฆื ืงืึธื ืืจืึธืืืจื, ืืึธืืืจ ืคึผืจืืืืจื ืืืึธืก ืืืืจืงืื ื ืื ืงืึทืืืืื ื ื ืืฆื ืคืื BlockCache ืึทืคึผืืึทืืึทืืืืฉืึทื ืืื SSC ืืื ืื ื ืคึฟืึทืจ ืืืืขื ืขื ืคึฟืื HBase ืืืขื ืืึธืื.
ืขืก ืงืขื ืืืื ืืขืืขื ืึทื ืืื ืืขืจ ืึทืืึท ืื ืึธืื ืื ืืืืจืงืื ื ืืื ื ืืฉื ืืื ืืจืืืก ืืื ืืื ืจืึทืคืื ืืจื ืืขืกืฅ (ืืืืขื ืขื ืึธื ืงืืื ืคึผืจืึทืกืขืกืื ื), ืึธืืขืจ ืขืก ืืื ืืึทื ืฅ ืืขืืืขื ืฆื ืงืืืขืืฉื ื ืึธื 80K ืืึธ. ืฆืืืึทืืขื, ืืืืืข ืึธืคึผืืืืืืึทืืืึธื ืก ืฆืืฉืืขืื ืึทืจืืืฃ ืฆื 4 ืงืก ืกืคึผืืืึทืคึผ.
ื ืคึผืจ ืืื ืืืื ืืขืืืื ืคึฟืึทืจ ืืขื ืึทืคึผืืึทืืึทืืืืฉืึทื
ืืื ืืขืกืึธืฃ, ืขืก ืืื ืืขืืืขื ืืฉืืงืึทืืืข ืฆื ืคืึทืจืืืืึทืื ืื ืืืืขื ืขื ืคืึธืจืฉืืขืืื ื ืคืื ืึท ืขื ืืขื ืืจืืื-ืืืึทื ืืึทืืึทืืืืก, Cassandra ืืื HBase.
ืฆื ืืึธื ืืึธืก, ืืืจ ืืึธื ืืฉื ืื ืกืืึทื ืกืื ืคืื ืื ื ืึธืจืืึทื YCSB ืืึทืกืข ืืขืกืืื ื ื ืืฆื ืคืื ืฆืืืื ืืื ืืช (800 ืคึฟืขืืขื ืืื ืืึทื ืฅ). ืืืืฃ ืื ืกืขืจืืืขืจ ืืืึทื - 4 ืื ืกืืึทื ืกืื ืคืื RegionServer ืืื Cassandra ืืืืฃ 4 ืืื ืืช (ื ืืฉื ืื ืืืึธืก ืื ืงืืืืึทื ืฅ ืืืืคื, ืฆื ืืืกืืืืื ืืืืขืจ ืืฉืคึผืขื). ืจืืืื ืื ืืขืงืืืขื ืคืื ืืืฉื ืคืื ืืจืืืก:
HBase - 300 GB ืืืืฃ HDFS (100 GB ืจืืื ืืึทืื)
ืงืึทืกืกืึทื ืืจืึท - 250 ืืืืืืืื (ืจืขืคึผืืึทืงืืืฉืึทื ืคืึทืงืืึธืจ = 3)
ืืขื ืข. ืืขืจ ืืึทื ื ืืื ืืขืืืขื ืืขืขืจืขื ืืขืจ ืืขืืืืงืขืจ (ืืื HBase ืึท ืืืกื ืืขืจ).
HBase ืคึผืึทืจืึทืืขืืขืจืก:
dfs.client.short.circuit.num = 5 (HDFS ืงืืืขื ื ืึทืคึผืืึทืืึทืืืืฉืึทื)
hbase.lru.cache.heavy.eviction.count.limit = 30 - ืืึธืก ืืืื ืึทื ืื ืืึทืืข ืืืขื ืึธื ืืืืื ืืจืืขืื ื ืึธื 30 ืืืืืงืฉืึทื ื (~ 5 ืืื ืื)
hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - ืฆืื ืืึทื ื ืคืื ืงืึทืืฉืื ื ืืื ืืืืืงืฉืึทื
YCSB ืืึธืืก ืืขื ืขื ืคึผืึทืจืกืขื ืืื ืงืึทืืคึผืืืื ืืื ืขืงืกืกืขื ืืจืึทืคืก:
ืืื ืืืจ ืงืขื ืขื ืืขื, ืื ืึธืคึผืืืืืืึทืืืึธื ืก ืืึทืื ืขืก ืืขืืืขื ืฆื ืคืึทืจืืืืึทืื ืื ืคืึธืจืฉืืขืืื ื ืคืื ืื ืืึทืืึทืืืืกืื ืืื ืืขืจ ืื ืืืืื ืืื ืืขื ืืื ืืขืจืืจืืืื 450 ืืืืื ื ืืืืขื ืขื ืคึผืขืจ ืกืขืงืื ืืข.
ืืืจ ืืึธืคื ืึทื ืื ืืื ืคึฟืึธืจืืึทืฆืืข ืงืขื ืืืื ื ืืฆืืง ืคึฟืึทืจ ืขืืขืฆืขืจ ืืขืฉืึทืก ืื ืืงืกืืืืื ื ืืขืจืึทื ืื ืคึฟืึทืจ ืคึผืจืึธืืืึทืงืืืืืืื.
ืืงืืจ: www.habr.com