RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMe

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMe
ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ ืžื™ืจ ื•ื•ืขืœืŸ ืจืขื“ืŸ ื•ื•ืขื’ืŸ ืคืึทืจืฉื™ื“ืขื ืข ื•ื•ืขื’ืŸ ืฆื• ืึธืจื’ืึทื ื™ื–ื™ืจืŸ RAID ืขืจื™ื™ื–, ืื•ืŸ ืื•ื™ืš ื•ื•ื™ื™ึทื–ืŸ ืื™ื™ื ืขืจ ืคื•ืŸ ื“ื™ ืขืจืฉื˜ืข ื™ื™ึทื–ื ื•ื•ืึทืจื’ RAID ืงืึทื ื˜ืจืึธื•ืœืขืจื– ืžื™ื˜ NVMe ืฉื˜ื™ืฆืŸ.

ืึทืœืข ื“ื™ ืคืึทืจืฉื™ื™ื“ื ืงื™ื™ึทื˜ ืคื•ืŸ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ืคื•ืŸ RAID ื˜ืขื›ื ืึธืœืึธื’ื™ืข ืื™ื– ื’ืขืคึฟื•ื ืขืŸ ืื™ืŸ ื“ื™ ืกืขืจื•ื•ืขืจ ืึธืคึผืฉื ื™ื˜. ืื™ืŸ ื“ืขื ืงืœื™ืขื ื˜ ืึธืคึผืฉื ื™ื˜, ื‘ืœื•ื™ื– ื•ื•ื™ื™ื›ื•ื•ืืจื’ RAID0 ืึธื“ืขืจ RAID1 ืื•ื™ืฃ ืฆื•ื•ื™ื™ ื“ื™ืกืงืก ืื™ื– ืžืขืจืกื˜ ืึธืคื˜ ื’ืขื ื™ืฆื˜.

ื“ืขืจ ืึทืจื˜ื™ืงืœ ื•ื•ืขื˜ ืฆื•ืฉื˜ืขืœืŸ ืึท ืงื•ืจืฅ ืื™ื‘ืขืจื‘ืœื™ืง ืคื•ืŸ RAID ื˜ืขื›ื ืึธืœืึธื’ื™ืข, ืึท ืงื•ืจืฅ ื˜ื•ื˜ืึธืจื™ืึทืœ ื•ื•ืขื’ืŸ ื•ื•ื™ ืฆื• ืฉืึทืคึฟืŸ RAID ืขืจื™ื™ื– ืžื™ื˜ ื“ืจื™ื™ ืคืึทืจืฉื™ื“ืขื ืข ืžื›ืฉื™ืจื™ื, ืื•ืŸ ืึท ืคืึทืจื’ืœื™ื™ึทืš ืคื•ืŸ ื•ื•ื™ืจื˜ื•ืึทืœ ื“ื™ืกืง ืคืึธืจืฉื˜ืขืœื•ื ื’ ืžื™ื˜ ื™ืขื“ืขืจ ืื•ืคึฟืŸ.

ื•ื•ืึธืก ืื™ื– RAID?

ื•ื•ื™ืงื™ืคึผืขื“ื™ืข ื’ื™ื˜ ืึท ืคื•ืœืฉื˜ืขื ื“ื™ืง ื“ืขืคึฟื™ื ื™ืฆื™ืข ืคื•ืŸ โ€‹โ€‹RAID ื˜ืขื›ื ืึธืœืึธื’ื™ืข:

ืึธื ืคืึทืœ (ืขื ื’ืœื™ืฉ ื™ื‘ืขืจื™ืง ืขืจื™ื™ ืคื•ืŸ ื™ื ื“ืขืคึผืขื ื“ืขื ื˜ ื“ื™ืกืงืก - ื™ื‘ืขืจื™ืง ืžืขื ื’ืข ืคื•ืŸ โ€‹โ€‹ืคืจื™ื™ึท (ืคืจื™ื™ึท) ื“ื™ืกืงืก) - ื“ืึทื˜ืŸ ื•ื•ื™ืจื˜ื•ืึทืœื™ื–ืึทื˜ื™ืึธืŸ ื˜ืขื›ื ืึธืœืึธื’ื™ืข ืคึฟืึทืจ ืงืึทืžื‘ื™ื™ื ื™ื ื’ ืขื˜ืœืขื›ืข ื’ืฉืžื™ื•ืช ื“ื™ืกืง ื“ืขื•ื•ื™ืกืขืก ืื™ืŸ ืึท ืœืึทื“ื–ืฉื™ืงืึทืœ ืžืึธื“ื•ืœืข ืฆื• ืคืึทืจื’ืจืขืกืขืจืŸ ืฉื•ืœื“ ื˜ืึธืœืขืจืึทื ืฅ ืื•ืŸ ืคืึธืจืฉื˜ืขืœื•ื ื’.

ื“ื™ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืคื•ืŸ ื“ื™ืกืง ืขืจื™ื™ื– ืื•ืŸ ื“ื™ ื˜ืขืงื ืึทืœืึทื“ื–ืฉื™ื– ื’ืขื ื™ืฆื˜ ืึธืคืขื ื’ืขืŸ ืื•ื™ืฃ ื“ื™ ืื•ื™ืกื’ืขืงืœื™ื‘ืŸ RAID ืžื“ืจื’ื”. RAID ืœืขื•ื•ืขืœืก ื–ืขื ืขืŸ ืกื˜ืึทื ื“ืขืจื“ื™ื™ื–ื“ ืื™ืŸ ื“ื™ ื‘ืึทืฉืจื™ื™ึทื‘ื•ื ื’ ืคึผืจืึธืกื˜ RAID ื“ื™ืกืง ื“ืึทื˜ืึท ืคึฟืึธืจืžืึทื˜. ืขืก ื‘ืืฉืจื™ื™ื‘ื˜ ืคื™ืœืข RAID ืœืขื•ื•ืขืœืก, ืึธื‘ืขืจ ื“ื™ ืžืขืจืกื˜ ืคึผืจืึธืกื˜ ื–ืขื ืขืŸ RAID0, RAID1, RAID5 ืื•ืŸ RAID6.

RAID0ืึธื“ืขืจ ืกื˜ืจื™ื™ืคึผืก, ืื™ื– ืึท RAID ืžื“ืจื’ื” ื•ื•ืึธืก ืงืึทืžื‘ื™ื™ื ื– ืฆื•ื•ื™ื™ ืึธื“ืขืจ ืžืขืจ ื’ืฉืžื™ื•ืช ื“ืจื™ื™ื•ื•ื– ืื™ืŸ ืื™ื™ืŸ ืœืึทื“ื–ืฉื™ืงืึทืœ ืคืึธืจ. ื“ืขืจ ื‘ืึทื ื“ ืคื•ืŸ ื“ื™ ืœืึทื“ื–ืฉื™ืงืึทืœ ื“ื™ืกืง ืื™ื– ื’ืœื™ื™ึทืš ืฆื• ื“ื™ ืกืึทื›ืึทืงืœ ืคื•ืŸ ื“ื™ ื•ื•ืึทืœื™ื•ืžื– ืคื•ืŸ ื“ื™ ื’ืฉืžื™ื•ืช ื“ื™ืกืงืก ืึทืจื™ื™ึทื ื’ืขืจืขื›ื ื˜ ืื™ืŸ ื“ื™ ืžืขื ื’ืข. ืขืก ืื™ื– ืงื™ื™ืŸ ื™ื‘ืขืจื™ืงื™ื™ึทื˜ ืื™ืŸ ื“ืขื RAID ืžื“ืจื’ื”, ืื•ืŸ ื“ื•ืจื›ืคืึทืœ ืคื•ืŸ ืื™ื™ืŸ ืคืึธืจ ืงืขื ืขืŸ ืจืขื–ื•ืœื˜ืึทื˜ ืื™ืŸ ื“ื™ ืึธื ื•ื•ืขืจ ืคื•ืŸ ืึทืœืข ื“ืึทื˜ืŸ ืื™ืŸ ื“ื™ ื•ื•ื™ืจื˜ื•ืึทืœ ื“ื™ืกืง.

ื’ืœื™ื™ึทืš RAID1ืึธื“ืขืจ ืฉืคึผื™ื’ืœ, ืงืจื™ื™ื™ืฅ ื™ื™ื“ืขื ื™ืงืึทืœ ืงืืคื™ืขืก ืคื•ืŸ ื“ืึทื˜ืŸ ืื•ื™ืฃ ืฆื•ื•ื™ื™ ืึธื“ืขืจ ืžืขืจ ื“ื™ืกืงืก. ื“ื™ ื’ืจื™ื™ืก ืคื•ืŸ ื“ื™ ื•ื•ื™ืจื˜ื•ืึทืœ ื“ื™ืกืง ื˜ื•ื˜ ื ื™ืฉื˜ ื™ืงืกื™ื“ ื“ื™ ืžื™ื ื™ืžื•ื ื’ืจื™ื™ืก ืคื•ืŸ ื“ื™ ื’ืฉืžื™ื•ืช ื“ื™ืกืงืก. ื“ื™ ื“ืึทื˜ืŸ ืื•ื™ืฃ ื“ื™ RAID1 ื•ื•ื™ืจื˜ื•ืึทืœ ื“ื™ืกืง ื•ื•ืขื˜ ื–ื™ื™ืŸ ื‘ืืจืขื›ื˜ื™ื’ื˜ ืึทื–ื•ื™ ืœืึทื ื’ ื•ื•ื™ ื‘ื™ื™ึท ืžื™ื ื“ืกื˜ืขืจ ืื™ื™ืŸ ื’ืฉืžื™ื•ืช ื“ื™ืกืง ืคื•ืŸ ื“ื™ ืžืขื ื’ืข ืื™ื– ืึทืคึผืขืจื™ื™ืฉืึทื ืึทืœ. ื ื™ืฆืŸ RAID1 ืžื•ืกื™ืฃ ื™ื‘ืขืจื™ืงื™ื™ึทื˜, ืึธื‘ืขืจ ืื™ื– ืึท ื’ืึทื ืฅ ื˜ื™ื™ึทืขืจ ืœื™ื™ื–ื•ื ื’, ื•ื•ื™ื™ึทืœ ืื™ืŸ ืขืจื™ื™ื– ืคื•ืŸ ืฆื•ื•ื™ื™ ืึธื“ืขืจ ืžืขืจ ื“ื™ืกืงืก ื“ื™ ืงืึทืคึผืึทืฆื™ื˜ืขื˜ ืคื•ืŸ ื‘ืœื•ื™ื– ืื™ื™ืŸ ืื™ื– ื‘ื ื™ืžืฆื.

ื’ืœื™ื™ึทืš RAID5 ืกืึทืœื•ื•ื– ื“ื™ ืคึผืจืึธื‘ืœืขื ืคื•ืŸ ื”ื•ื™ืš ืคึผืจื™ื™ึทื–. ืฆื• ืฉืึทืคึฟืŸ ืึท ืžืขื ื’ืข ืžื™ื˜ RAID5 ืžื“ืจื’ื”, ืื™ืจ ื“ืึทืจืคึฟืŸ ื‘ื™ื™ึท ืžื™ื ื“ืกื˜ืขืจ 3 ื“ื™ืกืงืก, ืื•ืŸ ื“ื™ ืžืขื ื’ืข ืื™ื– ืงืขื’ื ืฉื˜ืขืœื™ืง ืฆื• ื“ื™ ื“ื•ืจื›ืคืึทืœ ืคื•ืŸ ืื™ื™ืŸ ื“ื™ืกืง. ื“ืึทื˜ืŸ ืื™ืŸ RAID5 ื–ืขื ืขืŸ ืกื˜ืึธืจื“ ืื™ืŸ ื‘ืœืึทืงืก ืžื™ื˜ ื˜ืฉืขืงืกืึทืžื–. ืขืก ืื™ื– ืงื™ื™ืŸ ืฉื˜ืจืขื ื’ ืึธืคึผื˜ื™ื™ืœ ืฆื•ื•ื™ืฉืŸ ื“ืึทื˜ืŸ ื“ื™ืกืงืก ืื•ืŸ ื˜ืฉืขืงืงืกื•ื ื“ื™ืกืงืก. ื˜ืฉืขืงืงืกืึทืžื– ืื™ืŸ RAID5 ื–ืขื ืขืŸ ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ืคื•ืŸ ืึทืŸ XOR ืึธืคึผืขืจืึทืฆื™ืข ื’ืขื•ื•ืขื ื“ื˜ ืฆื• N-1 ื‘ืœืึทืงืก, ื™ืขื“ืขืจ ืคื•ืŸ ืึท ืึทื ื“ืขืจืฉ ื“ื™ืกืง.

ื›ืึธื˜ืฉ RAID ืขืจื™ื™ื– ืคืึทืจื’ืจืขืกืขืจืŸ ื™ื‘ืขืจื™ืงื™ื™ึทื˜ ืื•ืŸ ืฆื•ืฉื˜ืขืœืŸ ื™ื‘ืขืจื™ืงื™ื™ึทื˜, ื–ื™ื™ ื–ืขื ืขืŸ ื ื™ืฉื˜ ืคึผืึทืกื™ืง ืคึฟืึทืจ ืกื˜ืึธืจื™ื ื’ ื‘ืึทืงืึทืคึผืก.

ื ืึธืš ืึท ืงื•ืจืฅ ืฉืคึผืึทืฆื™ืจ ืื™ืŸ ื“ื™ ื˜ื™ื™ืคึผืก ืคื•ืŸ RAID ืขืจื™ื™ื–, ืื™ืจ ืงืขื ืขืŸ ืžืึทืš ืื•ื™ืฃ ืฆื• ื“ืขื•ื•ื™ืกืขืก ืื•ืŸ ืžื’ื™ืœื” ื•ื•ืึธืก ืœืึธื–ืŸ ืื™ืจ ืฆื• ืึทืกืขืžื‘ืึทืœ ืื•ืŸ ื ื•ืฆืŸ ื“ื™ืกืง ืขืจื™ื™ื–.

ื˜ื™ื™ืคึผืก ืคื•ืŸ RAID ืงืึทื ื˜ืจืึธื•ืœืขืจื–

ืขืก ื–ืขื ืขืŸ ืฆื•ื•ื™ื™ ื•ื•ืขื’ืŸ ืฆื• ืฉืึทืคึฟืŸ ืื•ืŸ ื ื•ืฆืŸ RAID ืขืจื™ื™ื–: ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืื•ืŸ ื•ื•ื™ื™ื›ื•ื•ืืจื’. ืžื™ืจ ื•ื•ืขืœืŸ ื‘ืึทื˜ืจืึทื›ื˜ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืกืึทืœื•ืฉืึทื ื–:

  • ืœื™ื ื•ืงืก ื•ื•ื™ื™ื›ื•ื•ืืจื’ RAID.
  • Intelยฎ Virtual RAID ืื•ื™ืฃ ืงืคึผื•.
  • LSI MegaRAID 9460-8i.

ื‘ืึทืžืขืจืงื•ื ื’ ืึทื– ื“ื™ Intelยฎ ืœื™ื™ื–ื•ื ื’ ืœื•ื™ืคื˜ ืื•ื™ืฃ ืึท ื˜ืฉื™ืคึผืกืขื˜, ื•ื•ืึธืก ืจื™ื™ื–ืึทื– ื“ื™ ืงืฉื™ื ืฆื™ ืขืก ืื™ื– ืึท ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืึธื“ืขืจ ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืœื™ื™ื–ื•ื ื’. ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ื“ื™ VMWare ESXi ื›ื™ื™ืคึผืขืจื•ื•ื™ื™ื–ืขืจ ื‘ืึทื˜ืจืึทื›ื˜ VROC ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืื•ืŸ ืฉื˜ื™ืฆื˜ ืขืก ื ื™ืฉื˜ ืึทืคื™ืฉืึทืœื™.

ืœื™ื ื•ืงืก ื•ื•ื™ื™ื›ื•ื•ืืจื’ RAID

ื•ื•ื™ื™ื›ื•ื•ืืจื’ RAID ืขืจื™ื™ื– ืื™ืŸ ื“ื™ ืœื™ื ื•ืงืก ืึทืก ืžืฉืคึผื—ื” ื–ืขื ืขืŸ ืึท ืคืขืจืœื™ ืคึผืจืึธืกื˜ ืœื™ื™ื–ื•ื ื’ ืื™ืŸ ื‘ื™ื™ื“ืข ืงืœื™ืขื ื˜ ืื•ืŸ ืกืขืจื•ื•ืขืจ ืกืขื’ืžืึทื ืฅ. ื›ืœ ืื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืฉืึทืคึฟืŸ ืึท ืžืขื ื’ืข ืื™ื– ื“ื™ mdadm ื ื•ืฆืŸ ืื•ืŸ ืขื˜ืœืขื›ืข ื‘ืœืึธืง ื“ืขื•ื•ื™ืกืขืก. ื“ืขืจ ื‘ืœื•ื™ื– ืคืึธื“ืขืจื•ื ื’ ืึทื– Linux Software RAID ืฉื˜ืขืœืŸ ืื•ื™ืฃ ื“ื™ ื“ืจื™ื™ื•ื•ื– ืขืก ื ื™ืฆื˜ ืื™ื– ืฆื• ื–ื™ื™ืŸ ืึท ื‘ืœืึธืง ืžื™ื˜ืœ ืฆื•ื˜ืจื™ื˜ืœืขืš ืฆื• ื“ื™ ืกื™ืกื˜ืขื.

ื“ืขืจ ืึทื•ื•ืขืง ืคื•ืŸ ืงืึธืก ืคึฟืึทืจ ื•ื™ืกืจื™ื›ื˜ ืื•ืŸ ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืื™ื– ืึท ืงืœืึธืจ ื•ื•ื™ ื“ืขืจ ื˜ืึธื’ ืžื™ื™ึทืœืข ืคื•ืŸ โ€‹โ€‹ื“ืขื ืื•ืคึฟืŸ. Linux Software RAID ืึธืจื’ืึทื ื™ื–ื™ืจื˜ ื“ื™ืกืง ืขืจื™ื™ื– ืื™ืŸ ื“ื™ ืคึผืจื™ื™ึทื– ืคื•ืŸ ืงืคึผื• ืฆื™ื™ื˜. ื“ื™ ืจืฉื™ืžื” ืคื•ืŸ ื’ืขืฉื˜ื™ืฆื˜ RAID ืœืขื•ื•ืขืœืก ืื•ืŸ ื“ื™ ืกื˜ืึทื˜ื•ืก ืคื•ืŸ ื“ื™ ืงืจืึทื ื˜ ื“ื™ืกืง ืขืจื™ื™ื– ืงืขื ืขืŸ ื–ื™ื™ืŸ ื•ื•ื™ื•ื“ ืื™ืŸ ื“ื™ mdstat ื˜ืขืงืข, ื•ื•ืึธืก ืื™ื– ืœื™ื’ืŸ ืื™ืŸ ื“ื™ ืคึผืจืึธืงืคืก ื•ื•ืึธืจืฆืœ:

root@grindelwald:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] 
unused devices: <none>

ืฉื˜ื™ืฆืŸ ืคึฟืึทืจ RAID ืœืขื•ื•ืขืœืก ืื™ื– ืžื•ืกื™ืฃ ื“ื•ืจืš ืงืึทื ืขืงื˜ื™ื ื’ ื“ื™ ืฆื•ื ืขืžืขืŸ ืงืขืจืŸ ืžืึธื“ื•ืœืข, ืœืžืฉืœ:

root@grindelwald:~# modprobe raid456
root@grindelwald:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
unused devices: <none>

ื›ืœ ืึทืคึผืขืจื™ื™ืฉืึทื ื– ืžื™ื˜ ื“ื™ืกืง ืขืจื™ื™ื– ื–ืขื ืขืŸ ื“ื•ืจื›ื’ืขืงืึธื›ื˜ ื“ื•ืจืš ื“ื™ mdadm ื‘ืึทืคึฟืขืœืŸ ืฉื•ืจื” ื ื•ืฆืŸ. ื“ื™ ื“ื™ืกืง ืžืขื ื’ืข ืื™ื– ืคืืจื–ืืžืœื˜ ืื™ืŸ ืื™ื™ืŸ ื‘ืึทืคึฟืขืœ:

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1

ื ืึธืš ืขืงืกืึทืงื™ื•ื˜ื™ื ื’ ื“ืขื ื‘ืึทืคึฟืขืœ, ื“ื™ / dev/md0 ื‘ืœืึธืง ืžื™ื˜ืœ ื•ื•ืขื˜ ื“ืขืจืฉื™ื™ึทื ืขืŸ ืื™ืŸ ื“ื™ ืกื™ืกื˜ืขื, ื•ื•ืึธืก ืจืขืคึผืจืึทื–ืขื ืฅ ืื™ืจ ื•ื•ื™ ืึท ื•ื•ื™ืจื˜ื•ืึทืœ ื“ื™ืกืง.

Intelยฎ Virtual RAID ืื•ื™ืฃ ืงืคึผื•

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeIntelยฎ VROC ื ืึธืจืžืึทืœ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืฉืœื™ืกืœ
Intelยฎ Virtual RAID On CPU (VROC) ืื™ื– ืึท ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืื•ืŸ ื•ื•ื™ื™ื›ื•ื•ืืจื’ ื˜ืขื›ื ืึธืœืึธื’ื™ืข ืคึฟืึทืจ ืงืจื™ื™ื™ื˜ื™ื ื’ RAID ืขืจื™ื™ื– ื‘ืื–ื™ืจื˜ ืื•ื™ืฃ Intelยฎ ื˜ืฉื™ืคึผืกืขืฅ. ื“ื™ ื˜ืขื›ื ืึธืœืึธื’ื™ืข ืื™ื– ื‘ื ื™ืžืฆื ื‘ืคึฟืจื˜ ืคึฟืึทืจ ืžืึธื˜ื”ืขืจื‘ืึธืึทืจื“ืก ื•ื•ืึธืก ืฉื˜ื™ืฆืŸ Intelยฎ Xeonยฎ ืกืงืึทืœืึทื‘ืœืข ืคึผืจืึทืกืขืกืขืจื–. ื“ื•ืจืš ืคืขืœื™ืงื™ื™ึทื˜, VROC ืื™ื– ื ื™ื˜ ื‘ื ื™ืžืฆื. ืฆื• ืึทืงื˜ืึทื•ื•ื™ื™ื˜ ืขืก, ืื™ืจ ืžื•ื–ืŸ ื™ื ืกื˜ืึทืœื™ืจืŸ ืึท VROC ื™ื™ึทื–ื ื•ื•ืึทืจื’ ื“ืขืจืœื•ื™ื‘ืขื ื™ืฉ ืฉืœื™ืกืœ.

ื“ืขืจ ื ืึธืจืžืึทืœ VROC ื“ืขืจืœื•ื™ื‘ืขื ื™ืฉ ืึทืœืึทื•ื– ืื™ืจ ืฆื• ืฉืึทืคึฟืŸ ื“ื™ืกืง ืขืจื™ื™ื– ืžื™ื˜ 0, 1 ืื•ืŸ 10 RAID ืœืขื•ื•ืขืœืก. ื“ื™ ืคึผืจืขืžื™ืข ื•ื•ืขืจืกื™ืข ื™ืงืกืคึผืึทื ื“ื– ื“ืขื ืจืฉื™ืžื” ืžื™ื˜ RAID5 ืฉื˜ื™ืฆืŸ.

Intelยฎ VROC ื˜ืขื›ื ืึธืœืึธื’ื™ืข ืื•ื™ืฃ ืžืึธื“ืขืจืŸ ืžืึธื˜ื”ืขืจื‘ืึธืึทืจื“ืก ืึทืจื‘ืขื˜ ืื™ืŸ ืงืึทื ื“ื–ืฉืึทื ื’ืงืฉืึทืŸ ืžื™ื˜ ื“ื™ Intelยฎ Volume Management Device (VMD), ื•ื•ืึธืก ื’ื™ื˜ ื”ื™ื™ืก ื•ื™ืกื‘ื™ื™ึทื˜ืŸ ืคื™ื™ื™ืงื™ื™ื˜ ืคึฟืึทืจ NVMe ื“ืจื™ื™ื•ื•ื–.

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeIntelยฎ VROC ืกื˜ืึทื ื“ืึทืจื“ ืœื™ืกืขื ืกืข ืึทืจืจื™ื™ึทืก ื–ืขื ืขืŸ ืงืึทื ืคื™ื’ื™ืขืจื“ ื“ื•ืจืš ื“ื™ ืกืขื˜ืึทืคึผ ื•ื˜ื™ืœื™ื˜ื™ ื•ื•ืขืŸ ื“ื™ ืกืขืจื•ื•ืขืจ ืฉื™ืš. ืื•ื™ืฃ ื“ื™ ืงื•ื•ื™ื˜ืœ ืึทื•ื•ืึทื ืกื™ืจื˜ืข ื“ื™ Intelยฎ Virtual RAID ืื•ื™ืฃ ืงืคึผื• ื ื•ืžืขืจ ืื™ื– ืืจื•ื™ืก, ื•ื•ื• ืื™ืจ ืงืขื ืขืŸ ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ืกืง ืขืจื™ื™ื–.

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeืฉืึทืคึฟืŸ ืึท RAID1 ืžืขื ื’ืข ืื•ื™ืฃ ืฆื•ื•ื™ื™ ื“ืจื™ื™ื•ื•ื–
Intelยฎ VROC ื˜ืขื›ื ืึธืœืึธื’ื™ืข ื”ืื˜ ื–ื™ื™ืŸ ืื™ื™ื’ืขื ืข ื™ื™ืกื™ื– ืื™ืŸ ื“ื™ ืึทืจื‘ืœ. ื“ื™ืกืง ืขืจื™ื™ื– ื’ืขื‘ื•ื™ื˜ ืžื™ื˜ VROC ื–ืขื ืขืŸ ืงืึทืžืคึผืึทื˜ืึทื‘ืึทืœ ืžื™ื˜ Linux Software RAID. ื“ืขื ืžื™ื˜ืœ ืึทื– ื“ื™ ืฉื˜ืึทื˜ ืคื•ืŸ ื“ื™ ืขืจื™ื™ื– ืงืขื ืขืŸ ื–ื™ื™ืŸ ืžืึธื ื™ื˜ืึธืจืขื“ ืื™ืŸ /proc/mdstat ืื•ืŸ ืึทื“ืžื™ื ืึทืกื˜ืขืจื“ ื“ื•ืจืš mdadm. ื“ืขื "ืคื•ื ืงืฆื™ืข" ืื™ื– ืึทืคื™ืฉืึทืœื™ ื’ืขืฉื˜ื™ืฆื˜ ื“ื•ืจืš ื™ื ื˜ืขืœ. ื ืึธืš ืึทืกืขืžื‘ืึทืœื™ื ื’ RAID1 ืื™ืŸ ืกืขื˜ืึทืคึผ ื•ื˜ื™ืœื™ื˜ื™, ืื™ืจ ืงืขื ืขืŸ ืึธื‘ืกืขืจื•ื•ื™ืจืŸ ื“ื™ ืกื™ื ื’ืงืจืึทื ืึทื–ื™ื™ืฉืึทืŸ ืคื•ืŸ ื“ืจื™ื™ื•ื•ื– ืื™ืŸ ื“ื™ ืึทืก:

root@grindelwald:~# cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid1 nvme2n1[1] nvme1n1[0]
      1855832064 blocks super external:/md127/0 [2/2] [UU]
      [>....................]  resync =  1.3% (24207232/1855832064) finish=148.2min speed=205933K/sec
      
md127 : inactive nvme1n1[1](S) nvme2n1[0](S)
      10402 blocks super external:imsm
       
unused devices: <none>

ื‘ืึทืžืขืจืงื•ื ื’ ืึทื– ืื™ืจ ืงืขื ืขืŸ ื ื™ืฉื˜ ืึทืกืขืžื‘ืึทืœ ืขืจื™ื™ื– ืื•ื™ืฃ VROC ื ื™ืฆืŸ mdadm (ื“ื™ ืึทืกืขืžื‘ืึทืœื“ ืขืจื™ื™ื– ื•ื•ืขื˜ ื–ื™ื™ืŸ Linux SW RAID), ืึธื‘ืขืจ ืื™ืจ ืงืขื ืขืŸ ื˜ื•ื™ืฉืŸ ื“ื™ืกืงืก ืื™ืŸ ื–ื™ื™ ืื•ืŸ ื“ื™ืกืึทืกืขืžื‘ืึทืœ ื“ื™ ืขืจื™ื™ื–.

LSI MegaRAID 9460-8i

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeืื•ื™ืกื–ืขืŸ ืคื•ืŸ ื“ื™ LSI MegaRAID 9460-8i ืงืึธื ื˜ืจืึธืœืœืขืจ
ื“ื™ RAID ืงืึธื ื˜ืจืึธืœืœืขืจ ืื™ื– ืึท ืฉื˜ื™ื™ืŸ-ืึทืœื™ื™ืŸ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืœื™ื™ื–ื•ื ื’. ื“ืขืจ ืงืึธื ื˜ืจืึธืœืœืขืจ ืึทืจื‘ืขื˜ ื‘ืœื•ื™ื– ืžื™ื˜ ื“ืจื™ื™ื•ื•ื– ืคืืจื‘ื•ื ื“ืŸ ื’ืœื™ื™ึทืš ืฆื• ืื™ื. ื“ืขืจ RAID ืงืึธื ื˜ืจืึธืœืœืขืจ ืฉื˜ื™ืฆื˜ ืึทืจื•ื™ืฃ ืฆื• 24 NVMe ื“ืจื™ื™ื•ื•ื–. ืขืก ืื™ื– ื“ื™ NVMe ืฉื˜ื™ืฆืŸ ื•ื•ืึธืก ืฉื˜ืขืœื˜ ื“ืขื ืงืึธื ื˜ืจืึธืœืœืขืจ ื‘ืึทื–ื•ื ื“ืขืจ ืคื•ืŸ ืคื™ืœืข ืื ื“ืขืจืข.

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeื”ื•ื™ืคึผื˜ ืžืขื ื™ื• ืคื•ืŸ ื“ื™ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืงืึธื ื˜ืจืึธืœืœืขืจ
ื•ื•ืขืŸ ืื™ืจ ื ื•ืฆืŸ UEFI ืžืึธื“ืข, ื“ื™ ืงืึธื ื˜ืจืึธืœืœืขืจ ืกืขื˜ื˜ื™ื ื’ืก ื–ืขื ืขืŸ ื™ื ืึทื’ืจื™ื™ื˜ื™ื“ ืื™ืŸ ื“ื™ ืกืขื˜ืึทืคึผ ื•ื˜ื™ืœื™ื˜ื™. ืงืึทืžืคึผืขืจื“ ืฆื• VROC, ื“ื™ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืงืึธื ื˜ืจืึธืœืœืขืจ ืžืขื ื™ื• ืงื•ืงื˜ ืคื™ืœ ืžืขืจ ืงืึธืžืคึผืœื™ืฆื™ืจื˜.

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMeืฉืึทืคึฟืŸ RAID1 ืื•ื™ืฃ ืฆื•ื•ื™ื™ ื“ื™ืกืงืก
ื“ืขืจืงืœืขืจืŸ ื•ื•ื™ ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ืกืง ืขืจื™ื™ื– ืื•ื™ืฃ ืึท ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืงืึธื ื˜ืจืึธืœืœืขืจ ืื™ื– ืึท ื’ืึทื ืฅ ื™ื™ื“ืœ ื˜ืขืžืข ืื•ืŸ ืงืขืŸ ื–ื™ื™ืŸ ื“ื™ ืกื™ื‘ื” ืคึฟืึทืจ ืึท ืคื•ืœ-ืคืœืขื“ื–ืฉื“ ืึทืจื˜ื™ืงืœ. ื“ืึธ ืžื™ืจ ื•ื•ืขืœืŸ ืคืฉื•ื˜ ื‘ืึทื’ืจืขื ืขืฆืŸ ื–ื™ืš ืฆื• ืงืจื™ื™ื™ื˜ื™ื ื’ RAID0 ืื•ืŸ RAID1 ืžื™ื˜ ืคืขืœื™ืงื™ื™ึทื˜ ืกืขื˜ื˜ื™ื ื’ืก.

ื“ื™ืกืงืก ืงืึธื ื ืขืงื˜ืขื“ ืฆื• ื“ื™ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืงืึธื ื˜ืจืึธืœืœืขืจ ื–ืขื ืขืŸ ื ื™ืฉื˜ ืงืขื ื˜ื™ืง ืฆื• ื“ื™ ืึธืคึผืขืจื™ื™ื˜ื™ื ื’ ืกื™ืกื˜ืขื. ืึทื ืฉื˜ืึธื˜, ื“ื™ ืงืึธื ื˜ืจืึธืœืœืขืจ "ืžืึทืกืง" ืึทืœืข RAID ืขืจื™ื™ื– ื•ื•ื™ SAS ื“ืจื™ื™ื•ื•ื–. ื“ืจื™ื™ื•ื•ื– ืงืึธื ื ืขืงื˜ืขื“ ืฆื• ื“ื™ ืงืึธื ื˜ืจืึธืœืœืขืจ, ืึธื‘ืขืจ ื ื™ืฉื˜ ื˜ื™ื™ืœ ืคื•ืŸ ื“ื™ ื“ื™ืกืง ืžืขื ื’ืข, ื•ื•ืขื˜ ื ื™ืฉื˜ ื–ื™ื™ืŸ ืฆื•ื˜ืจื™ื˜ืœืขืš ื“ื•ืจืš ื“ื™ ืึทืก.

root@grindelwald:~# smartctl -i /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               AVAGO
Product:              MR9460-8i
Revision:             5.14
Compliance:           SPC-3
User Capacity:        1,999,844,147,200 bytes [1.99 TB]
Logical block size:   512 bytes
Rotation Rate:        Solid State Device
Logical Unit id:      0x000000000000000000000000000000
Serial number:        00000000000000000000000000000000
Device type:          disk
Local Time is:        Sun Oct 11 16:27:59 2020 MSK
SMART support is:     Unavailable - device lacks SMART capability.

ื˜ืจืึธืฅ ื–ื™ื™ืŸ ื“ื™ืกื’ื™ื™ื–ื“ ื•ื•ื™ SAS ื“ืจื™ื™ื•ื•ื–, NVMe ืขืจื™ื™ื– ื•ื•ืขื˜ ืึทืจื‘ืขื˜ืŸ ืžื™ื˜ PCIe ืกืคึผื™ื“ื–. ืึธื‘ืขืจ, ื“ืขื ืฉื˜ืจื™ืš ืึทืœืึทื•ื– ืื™ืจ ืฆื• ืฉื˜ื™ื•ื•ืœ ืคึฟื•ืŸ NVMe ืื™ืŸ Legacy.

ื˜ืขืกื˜ ืฉื˜ื™ื™ืŸ

ื™ืขื“ืขืจ ืื•ืคึฟืŸ ืคื•ืŸ ืึธืจื’ืึทื ื™ื™ื–ื™ื ื’ ื“ื™ืกืง ืขืจื™ื™ื– ื”ืื˜ ื–ื™ื™ึทืŸ ืื™ื™ื’ืขื ืข ื’ืฉืžื™ื•ืช ืคึผืจืึธืก ืื•ืŸ ืงืึธื ืก. ืึธื‘ืขืจ ืื™ื– ืขืก ืึท ืคืึธืจืฉื˜ืขืœื•ื ื’ ื—ื™ืœื•ืง ื•ื•ืขืŸ ืืจื‘ืขื˜ืŸ ืžื™ื˜ ื“ื™ืกืง ืขืจื™ื™ื–?

ืฆื• ื“ืขืจื’ืจื™ื™ื›ืŸ ืžืึทืงืกื™ืžื•ื ื™ื•ื™ืฉืขืจ, ืึทืœืข ื˜ืขืกืฅ ื•ื•ืขื˜ ื–ื™ื™ืŸ ื’ืขืคื™ืจื˜ ืื•ื™ืฃ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืกืขืจื•ื•ืขืจ. ื–ื™ื™ึทืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ:

  • 2x Intelยฎ Xeonยฎ 6240;
  • 12 ืงืก ื“ื“ืจ4-2666 16 ื’ื™ื’ืื‘ื™ื™ื˜;
  • LSI MegaRAID 9460-8i;
  • Intelยฎ VROC Standard Hardware Key;
  • 4x Intelยฎ SSD DC P4510 U.2 2TB;
  • 1x Samsung 970 EVO Plus M.2 500GB.

ื“ื™ ืคึผืจืึธื‘ืข ื•ื ื™ืฅ ื–ืขื ืขืŸ P4510, ืื™ื™ื ืขืจ ื”ืึทืœื‘ ืคื•ืŸ ื•ื•ืึธืก ืื™ื– ืงืึธื ื ืขืงื˜ืขื“ ืฆื• ื“ื™ ืžืึธื˜ื”ืขืจื‘ืึธืึทืจื“, ืื•ืŸ ื“ื™ ืื ื“ืขืจืข ื”ืขืœืคื˜ ืฆื• ื“ื™ RAID ืงืึธื ื˜ืจืึธืœืœืขืจ. ื“ื™ M.2 ืื™ื– ืคืœื™ืกื ื“ื™ืง Ubuntu 20.04 ืื•ืŸ ื“ื™ ื˜ืขืกืฅ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืœื•ื™ืคืŸ ืžื™ื˜ ืคื™ืึธ ื•ื•ืขืจืกื™ืข 3.16.

ื˜ืขืกื˜ื™ื ื’

ืขืจืฉื˜ืขืจ ืคื•ืŸ ืึทืœืข, ืœืึธื–ืŸ ืก ืงืึธื ื˜ืจืึธืœื™ืจืŸ ื“ื™ ื“ื™ืœื™ื™ื– ื•ื•ืขืŸ ืืจื‘ืขื˜ืŸ ืžื™ื˜ ื“ื™ ื“ื™ืกืง. ื“ื™ ืคึผืจืึธื‘ืข ืื™ื– ืขืงืกืึทืงื™ื•ื˜ืึทื“ ืื™ืŸ ืื™ื™ืŸ ืคืึธื“ืขื, ื“ื™ ื‘ืœืึธืง ื’ืจื™ื™ืก ืื™ื– 4 ืงื‘. ื™ืขื“ืขืจ ืคึผืจืึธื‘ืข ืœืึทืกืฅ 5 ืžื™ื ื•ื˜. ืื™ื™ื“ืขืจ ืกื˜ืึทืจื˜ื™ื ื’, ื“ื™ ืงืึธืจืึทืกืคึผืึทื ื“ื™ื ื’ ื‘ืœืึธืง ืžื™ื˜ืœ ืื™ื– ื‘ืึทืฉื˜ื™ืžื˜ ืฆื• ื’ืึธืจื ื™ื˜ ื•ื•ื™ ื“ื™ I / O ืกืงืขื“ื–ืฉื•ืœืขืจ. ื“ื™ fio ื‘ืึทืคึฟืขืœ ืงื•ืงื˜ ื•ื•ื™ ื“ืึธืก:

fio --name=test --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio  --iodepth=1 --loops=1000 --runtime=300  --rw=<mode> --filename=<blkdev>

ืคึฟื•ืŸ ืคื™ืึธ ืจืขื–ื•ืœื˜ืึทื˜ืŸ ืžื™ืจ ื ืขืžืขืŸ ืงืœืึทื˜ 99.00%. ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ืŸ ื–ืขื ืขืŸ ื’ืขื•ื•ื™ื–ืŸ ืื™ืŸ ื“ื™ ื˜ื™ืฉ ืื•ื ื˜ืŸ.

ื˜ืจืึทืค - ืœื™ื™ืขื ืขืŸ, ฮผs
ื˜ืจืึทืค ืจืขืงืึธืจื“ื™ื ื’, ฮผs

ื“ื™ืกืง
112
78

Linux SW RAID, RAID0
113
45

VROC, RAID0
112
46

LSI, RAID0
122
63

Linux SW RAID, RAID1
113
48

VROC, RAID1
113
45

LSI, RAID1
128
89

ืื™ืŸ ืึทื“ื™ืฉืึทืŸ ืฆื• ื“ื™ืœื™ื™ื– ื•ื•ืขืŸ ืึทืงืกืขืกื™ื ื’ ื“ืึทื˜ืŸ, ืื™ืš ื•ื•ืึธืœื˜ ื•ื•ื™ ืฆื• ื–ืขืŸ ื“ื™ ืคืึธืจืฉื˜ืขืœื•ื ื’ ืคื•ืŸ ื•ื•ื™ืจื˜ื•ืึทืœ ื“ืจื™ื™ื•ื•ื– ืื•ืŸ ืคืึทืจื’ืœื™ื™ึทื›ืŸ ื–ื™ื™ ืžื™ื˜ ื“ื™ ืคืึธืจืฉื˜ืขืœื•ื ื’ ืคื•ืŸ ืึท ื’ืฉืžื™ื•ืช ื“ื™ืกืง. ื‘ืึทืคึฟืขืœ ืฆื• ืœื•ื™ืคืŸ fio:

fio --name=test --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio  --loops=1000 --runtime=300  --iodepth=<threads> --rw=<mode> --filename=<blkdev>

ืคืึธืจืฉื˜ืขืœื•ื ื’ ืื™ื– ื’ืขืžืืกื˜ืŸ ืื™ืŸ ื˜ืขืจืžื™ื ืขืŸ ืคื•ืŸ ื™ / ืึธ ืึทืคึผืขืจื™ื™ืฉืึทื ื–. ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ืŸ ื–ืขื ืขืŸ ื“ืขืจืœืื ื’ื˜ ืื™ืŸ ื“ื™ ื˜ื™ืฉ ืื•ื ื˜ืŸ.

ื˜ืจืึทืค ืœื™ื™ืขื ืขืŸ 1 ืคืึธื“ืขื, IOPS
ืจืึทื ื“ืึธื ืฉืจื™ื™ึทื‘ืŸ 1 ืคืึธื“ืขื, IOPS
ื˜ืจืึทืค ืœื™ื™ืขื ืขืŸ 128 ืคึฟืขื“ืขื, IOPS
ืจืึทื ื“ืึธื ืฉืจื™ื™ึทื‘ืŸ 128 ืคึฟืขื“ืขื, IOPS

ื“ื™ืกืง
11300
40700
453000
105000

Linux SW RAID, RAID0
11200
52000
429000
232000

VROC, RAID0
11200
52300
441000
162000

LSI, RAID0
10900
44200
311000
160000

Linux SW RAID, RAID1
10000
48600
395000
147000

VROC, RAID1
10000
54400
378000
244000

LSI, RAID1
11000
34300
229000
248000

ืขืก ืื™ื– ื’ืจื™ื ื’ ืฆื• ื–ืขืŸ ืึทื– ื ื™ืฆืŸ ืึท ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืงืึทื ื˜ืจืึธื•ืœืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ืื™ืŸ ืึท ื’ืขื•ื•ืืงืกืŸ ืœื™ื™ื˜ืึทื ืกื™ ืื•ืŸ ืึท ืคืึธืจืฉื˜ืขืœื•ื ื’ ืฉืœืึธื’ืŸ ืงืึทืžืคึผืขืจื“ ืžื™ื˜ ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืกืึทืœื•ืฉืึทื ื–.

ืกืึธืฃ

ื ื™ืฆืŸ ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืกืึทืœื•ืฉืึทื ื– ืฆื• ืฉืึทืคึฟืŸ ื“ื™ืกืง ืขืจื™ื™ื– ืคื•ืŸ ืฆื•ื•ื™ื™ ื“ื™ืกืงืก ืงื•ืงื˜ ื™ืจืึทืฉืึทื ืึทืœ. ืึธื‘ืขืจ, ืขืก ื–ืขื ืขืŸ ื˜ืึทืกืงืก ื•ื•ื• ื“ื™ ื ื•ืฆืŸ ืคื•ืŸ RAID ืงืึทื ื˜ืจืึธื•ืœืขืจื– ืื™ื– ื’ืขืจืขื›ื˜ืคืืจื˜ื™ืงื˜. ืžื™ื˜ ื“ื™ ืึทื“ื•ื•ืขื ื˜ ืคื•ืŸ ืงืึทื ื˜ืจืึธื•ืœืขืจื– ื•ื•ืึธืก ืฉื˜ื™ืฆืŸ ื“ื™ NVMe ืฆื•ื‘ื™ื ื“, ื™ื•ื–ืขืจื– ื”ืึธื‘ืŸ ื“ื™ ื’ืขืœืขื’ื ื”ื™ื™ื˜ ืฆื• ื ื•ืฆืŸ ืคืึทืกื˜ืขืจ ืกืกื“ืก ืื™ืŸ ื–ื™ื™ืขืจ ืคึผืจืึทื“ื–ืฉืขืงืก.

RAID ืขืจื™ื™ื– ืื•ื™ืฃ NVMe

ื‘ืœื•ื™ื– ืจืขื’ื™ืกื˜ืจื™ืจื˜ ื ื™ืฆืขืจืก ืงืขื ืขืŸ ืึธื ื˜ื™ื™ืœ ื ืขืžืขืŸ ืื™ืŸ ื“ื™ ื™ื‘ืขืจื‘ืœื™ืง. ืกื™ื™ืŸ ืืจื™ื™ืŸื‘ื™ื˜ืข.

ื–ืขื ื˜ ืื™ืจ ื ื™ืฆืŸ RAID ืกืึทืœื•ืฉืึทื ื–?

  • ืงืกื ื•ืžืงืก%ื™ืึธ, ื™ื™ึทื–ื ื•ื•ืึทืจื’ ืกืึทืœื•ืฉืึทื ื–32

  • ืงืกื ื•ืžืงืก%ื™ืึธ, ื•ื•ื™ื™ื›ื•ื•ืืจื’ ืกืึทืœื•ืฉืึทื ื–54

  • ืงืกื ื•ืžืงืก%No18

  • ืงืกื ื•ืžืงืก%ืงื™ื™ืŸ RAID ื“ืืจืฃ 4

108 ื‘ืื ื•ืฆืขืจ ื”ืื‘ืŸ ื’ืขืฉื˜ื™ืžื˜. 14 ื‘ืื ื•ืฆืขืจ ื”ืื‘ืŸ ื–ื™ืš ืืคื’ืขื”ืืœื˜ืŸ.

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’