ืืื ืฆื ืืืืขื ืขื ืืขื ืึทืจืืืงื: ืืื ืึทื ืืฉืืืืืงื ืคึฟืึทืจ ืื ืืขืงืกื ืืื ืึทืืื ืืึทื ื ืืื ืืึทืึธืืืฉ. ืึผืื ืฆื ืฉืคึผืึธืจื ืฆืืื, ืืื ืึธื ืืืืื ืืขืืขืจ ืงืึทืคึผืืื ืืื ืึท ืืงืืื "ืืืึธืก ืืื ืืขืืขืจื ื", ืืืึธืก ืกืึทืืขืจืืืืื ืื ืขืกืึทื ืก ืคืื ืืขื ืงืึทืคึผืืื ืืื ืืืื ืึธืืขืจ ืฆืืืื ืืืฆื.
"ื ืึธืจ ืืืืึทืื ืืืจ ืื ืืืืืื ื!" ืืืื ืืืจ ื ืึธืจ ืืืืื ืฆื ืืขื ืืื ืืื ืืขืงืืืขื ืคึฟืื, ืืึทื ืืึธืคึผืงืขื ืฆื ืื ืงืึทืคึผืืื "ืืืขืจื ืืขืจ ืื ืืืขื ืืืื," ืึธืืขืจ ืืื ืืจืึทืืื ืขืก ืืื ืืขืจ ืืฉืืงืึทืืืข ืืื ื ืืฆืืง ืฆื ืืืืขื ืขื ืืืขืื ืืืจืืคืึทื.
ืืื ืืื ืืขืฆืื ืก ืืึทืกืงื ืืื ืืึทืฉืืขืืืงื ืึท ืคึผืจืึธืฆืขืก ืคึฟืึทืจ ืคึผืจืึทืกืขืกืื ื ืึท ืืจืืืก ืืึทื ื ืคืื ืจืื ืื ืึท ืกืืงืืืึทื ืกืื (ืืขืงื ืืงืื ืึท SNP ืฉืคึผืึธื). ืื ื ืืื ืืื ืืขืืืขื ืฆื ืืขืฉืืืื ื ืืึทืงืืืขื ืืึทืื ืืืขืื ืึท ืืขืืขืื ืืขื ืขืืืง ืึธืจื (ืืขืจืืคื ืึท SNP) ืคึฟืึทืจ ืกืึทืืกืึทืงืืืึทื ื ืืึธืืขืืื ื ืืื ืื ืืขืจืข ืืึทืกืงืก. ื ืืฆื R ืืื AWK, ืืื ืืื ืืขืืืขื ืืืืืืช ืฆื ืจืืื ืืื ืึธืจืืึทื ืืืืจื ืืึทืื ืืืืฃ ืึท ื ืึทืืืจืืขื ืืืขื, ืืื ืืืืขืจ ืคืึทืจืืืืขืจื ืึธื ืคึฟืจืขื ืคึผืจืึทืกืขืกืื ื. ืืึธืก ืืื ื ืืฉื ืืจืื ื ืคึฟืึทืจ ืืืจ ืืื ืคืืจืืื ืื ืคืืืข ืืืขืจืืืฉืึทื ื. ืืขืจ ืึทืจืืืงื ืืืขื ืืขืืคึฟื ืืืจ ืืืกืืืืื ืขืืืขืืข ืคืื โโโโืืืื ืืืกืืืืงืก ืืื ืืืืึทืื ืืืจ ืืืึธืก ืืื ืขื ืืืงื ืืื ืืื.
ืขืจืฉืืขืจ, ืขืืืขืืข ืื ืืจืึทืืึทืงืืขืจื ืืขืจืงืืขืจืื ืืขื.
ืืึทืืข
ืืื ืืืขืจ ืืื ืืืืขืจืกืืืขื ืืขื ืขืืืง ืืื ืคึฟืึธืจืืึทืฆืืข ืคึผืจืึทืกืขืกืื ื ืฆืขื ืืขืจ ืฆืืืขืฉืืขืื ืืื ืื ืืื ืืึทืื ืืื ืื ืคืึธืจืขื ืคืื ืึท 25 TB TSV. ืืื ืืืงืืืขื ืืื ืฆืขืืืืื ืืื 5 ืคึผืึทืงืึทืืืฉืึทื, ืงืึทืืคึผืจืขืกื ืืืจื Gzip, ืืขืืขืจ ืคืื ืืืึธืก ืึผืืื ืืืขืื 240 ืคืืจ-ืืืืืืืื ืืขืงืขืก. ืืขืืขืจ ืจืืืขืจื ืงืึทื ืืืื ื ืืึทืื ืคึฟืึทืจ ืืืื SNP ืคืื ืืืื ืืืื. ืืื ืืึทื ืฅ, ืืึทืื ืืืขืื ~ 2,5 ืืืืืึธื ืกื ืคึผืก ืืื ~ 60 ืืืืื ื ืืขื ืืฉื ืืขื ืขื ืืจืึทื ืกืืืืืขื. ืืื ืึทืืืฉืึทื ืฆื SNP ืืื ืคึฟืึธืจืืึทืฆืืข, ืื ืืขืงืขืก ืงืึทื ืืืื ื ืคืืืข ืฉืคืืืื ืืื ื ืืืขืจื ืืืึธืก ืจืืคืืขืงืืื ื ืคืึทืจืฉืืื ืงืขืจืึทืงืืขืจืืกืืืงืก, ืึทืืึท ืืื ืืืืขื ืขื ืื ืืขื ืกืืื, ืึธืคืืงืืึทื ืคืื ืคืึทืจืฉืืืขื ืข ืึทืืืขืก, ืืื"ื ื. ืืื ืืึทื ืฅ ืขืก ืืขื ืขื ืืขืืืขื ืืืขืื 30 ืฉืคืืืื ืืื ืืื ืฆืืง ืืืึทืืืขืก.
ืฆืื
ืืื ืืื ืงืืื ืืึทืื ืคืึทืจืืืึทืืืื ื ืคึผืจืืืขืงื, ืื ืืขืจืกื ืืืืืืืง ืืึทื ืืื ืืขืืืขื ืฆื ืืึทืฉืืืกื ืืื ืื ืืึทืื ืืืึธืื ืืืื ืืขืืืืื ื. ืืื ืืขื ืคืึทื ืืืจ ืืืขืื ืืขืจืกืื ืก ืืืืกืงืืืึทืื ืืึธืืขืืก ืืื ืืืึธืจืงืคืืึธืื ืคึฟืึทืจ SNP ืืืืืจื ืืืืฃ SNP. ืืึธืก ืืื, ืืืจ ืืืขืื ืืืืื ืืึทืจืคึฟื ืืึทืื ืืืืฃ ืืืื SNP ืืื ืึท ืฆืืื. ืืื ืืื ืฆื ืืขืจื ืขื ืืื ืฆื ืฆืืจืืงืงืจืืื ืึทืืข ืื ืจืขืงืึธืจืืก ืคึฟืึทืจืืื ืื ืืื ืืืื ืขืจ ืคืื ืื 2,5 ืืืืืึธื ืกื ืคึผืก ืืื ืืืืื, ืืขืฉืืืื ื ืืื ืืืืืง ืืื ืืขืืืขื.
ืืื ื ืื ืฆื ืืึธื ืืึธืก
ืฆื ืฆืืืืจื ืึท ืคึผืึทืกืืงื ืงืืืืฉ:
ืืื ืืื ื ืืฉื ืืืจืืคืึทื ืึท ืืืืื ื ืืึธื, ืืื ื ืึธืจ ืืืกืงืึทืืืขืจื ืึท ืืืืื ื ืืืขืื ืฆื ืืืกืืืืื ืคึผืึทืจืกืื ื ืึท ืืื ืื ืคืื ืืึทืื ืืื ืึท ืึธื ืคึฟืจืขื-ืคืจืืึทื ืืืขื ืคึฟืึธืจืืึทื.
ืขืจืฉืืขืจ ืคึผืจืืืืจื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืขืก ืืื ืงืืื ืืืืืง ืืืขื ืฆื ืคึผืึทืจืกืืจื 25 ืื ืืื ืึท ืฆืืื.
ืืื ืืึธื ืืขื ืืืขื ืืขื ืงืืจืก "ืึทืืืึทื ืกืืจืืข ืืขืืืึธืืก ืคึฟืึทืจ ืืื ืืึทืืึท ืคึผืจืึทืกืขืกืื ื" ืืื ืืืึทื ืืขืจืืืื ืืื ืืืืขืจืกืืืขื, ืืื ืืื ืืขืืืขื ืืืืขืจ ืึทื ืืขืจ ืงืื ืฅ ืืื ืืื ืื ืืึทืฉ. ืขืก ืืืขื ืืืกืืึธืืข ื ืขืืขื ืึท ืฉืขื ืึธืืขืจ ืฆืืืื ืฆื ืฉืืขืื ืื ืืืืืข ืกืขืจืืืขืจ ืฆื ืืืืคื ืืืจื ืึทืืข ืื ืืึทืื ืืื ืืึทืจืืื ืื ืจืขืืืืืึทื. ืืื ื ืืื ืืืขืจ ืืึทืื ืืขื ืขื ืกืืึธืจื ืืื AWS S3, ืืื ืืขืืืืื ื ืื ืกืขืจืืืืก , ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืฆืืืืืื Hive SQL ืงืืืืจืื ืฆื S3 ืืึทืื. ืืืจ ืืึธื ื ืื ืืึทืจืคึฟื ืฆื ืฉืืขืื ืึทืจืืืฃ / ืืึทืคึผื ืึท ืืืืืข ืงื ืืื, ืืื ืืืจ ืืืื ืืึทืฆืึธืื ืืืืื ืคึฟืึทืจ ืื ืืึทืื ืืืจ ืืืื ืคึฟืึทืจ.
ื ืึธื ืืื ืืขืืืืื ืึทืืืขื ืึท ืืืื ืืึทืื ืืื ืืืื ืคึฟืึธืจืืึทื, ืืื ืืืืคื ืขืืืขืืข ืืขืกืฅ ืืื ืคึฟืจืืื ืืื ืืึธืก:
select * from intensityData limit 10;ืืื ืืขืฉืืืื ื ืืืงืืืขื ืืขืืื ื-ืกืืจืึทืงืืฉืขืจื ืจืขืืืืืึทืื. ืืจืืื.
ืืื ืืืจ ืืขืคืจืืืื ืฆื ื ืืฆื ืื ืืึทืื ืืื ืืื ืืืขืจ ืึทืจืืขื ...
ืืื ืืื ืืขืืืขื ืืขืืขืื ืฆื ืฆืืขื ืึทืืข ืื SNP ืืื ืคึฟืึธืจืืึทืฆืืข ืฆื ืคึผืจืืืืจื ืื ืืึธืืขื ืืืืฃ. ืืื ืืื ืืขืืืื ื ืื ืคืจืืืข:
select * from intensityData
where snp = 'rs123456';... ืืื ืื ืืขืืืืื ืฆื ืืืึทืจืื. ื ืึธื ืึทืื ืืื ืื ืืื ืืขืจ ืืื 4 ืื ืคืื ืืขืืขืื ืืึทืื, ืืื ืืืงืืืขื ืื ืจืขืืืืืึทื. ืึทืืืขื ืึท ืืฉืึทืจืืืฉืื ืืืื ืื ืืึทื ื ืคืื ืืึทืื ืืขืคึฟืื ืขื, $ 5 ืคึผืขืจ ืืขืจืึทืืืืข. ืึทืืื ืืขื ืืืื ืืงืฉื ืงืึธืก $ 20 ืืื ืึทืื ืืื ืื ืคืื ืืืึทืจืื. ืฆื ืืืืคื ืืขื ืืึธืืขื ืืืืฃ ืึทืืข ืื ืืึทืื, ืืืจ ืืึธืื ืฆื ืืืึทืจืื 38 ืืืจ ืืื ืืึทืฆืึธืื $ 50 ืืืืืึธื, ืืึธื, ืืึธืก ืืื ื ืืฉื ืคึผืึทืกืืง ืคึฟืึทืจ ืืื ืื.
ืขืก ืืื ื ืืืืืง ืฆื ื ืืฆื ืคึผืึทืจืงืื ...
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืืืื ืึธืคึผืืขืืื ืืื ืื ืืจืืืก ืคืื ืืืื ืคึผืึทืจืงืืืขื ืืขืงืขืก ืืื ืืืืขืจ ืึธืจืืึทื ืืืึทืฆืืข.
ืืื ืขืจืฉืืขืจ ืืขืคืจืืืื ืฆื ืคืึทืจืจืืืื ืื ืกืืืืึทืฆืืข ืืืจื ืงืึทื ืืืขืจืืื ื ืึทืืข TSVs ืฆื . ืืื ืืขื ืขื ืืึทืงืืืขื ืคึฟืึทืจ ืืจืืขืื ืืื ืืจืืืก ืืึทืื ืฉืืขืื ืืืืึทื ืื ืืื ืคึฟืึธืจืืึทืฆืืข ืืื ืืื ืืื ืกืืึธืจื ืืื ืงืึธืืืื ืขืจ ืคืึธืจืขื: ืืขืืขืจ ืืืึทื ืืืื ืืื ืืืื ืืืืืขื ืข ืืึผืจืื / ืืืกืง ืึธืคึผืฉื ืื, ืืื ืงืึทื ืืจืึทืกื ืฆื ืืขืงืกื ืืขืงืขืก, ืืื ืืืึธืก ืจืึธืื ืึทื ืืืึทืืื ืขืืขืืขื ืื ืคืื ืืขืืขืจ ืืืึทื. ืืื ืืืื ืืืจ ืืึทืจืคึฟื ืฆื ืืขืคึฟืื ืขื ืขืคึผืขืก, ื ืึธืจ ืืืืขื ืขื ืื ืคืืจืืื ืื ืืืึทื. ืึทืืืืืืึธื ืึทืืื, ืืขืืขืจ ืืขืงืข ืกืืึธืจื ืึท ืงืืื ืคืื ืืืึทืืืขืก ืืื ืึท ืืืึทื, ืึทืืื ืืืื ืื ืืืขืจื ืืืจ ืืืื ืคึฟืึทืจ ืืื ื ืืฉื ืืื ืื ืงืืื ืคืื ืื ืืืึทื, ืกืคึผืึทืจืง ืืืขื ื ืืฉื ืืืืกื ืฆืืื ืฆื ืืืขืจืงืืงื ืื ืืื ืฆืข ืืขืงืข.
ืืื ืืขืืืคื ืึท ืคึผืฉืื ืึทืจืืขื ืฆื ืืขืจ ืืื ืืืขืจ TSVs ืฆื ืคึผืึทืจืงืืืขื ืืื ืืจืึทืคึผื ืื ื ืืึทืข ืืขืงืขืก ืืื ืึทืืืขื ืึท. ืขืก ืืขื ืืืขื ืืืขืื 5 ืฉืขื. ืืืขืจ ืืืขื ืืื ืืืืคื ืื ืืงืฉื, ืขืก ืืขื ืืืขื ืืืขืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื ืืื ืึท ืืืกื ืืืืื ืืงืขืจ ืืขืื ืฆื ืคืึทืจืขื ืืืงื. ืืขืจ ืคืึทืงื ืืื ืึทื ืกืคึผืึทืจืง, ืืจืืื ื ืฆื ืึทืคึผืืึทืืืื ืื ืึทืจืืขื, ืคืฉืื ืึทื ืคึผืึทืงื ืืืื TSV ืคึผืืึทืืข ืืื ืฉืืขืื ืขืก ืืื ืืืื ืืืืื ืคึผืึทืจืงืื ืฉืืืง. ืืื ืืืืึทื ืืขืืขืจ ืฉืืืง ืืื ืืขืืืขื ืืจืืืก ืืขื ืื ืฆื ืึทื ืืืึทืืื ืื ืืื ืฆืข ืจืขืงืึธืจืืก ืคืื ืคืืืข ืืขื ืืฉื, ืืขืืขืจ ืืขืงืข ืึผืืื ืึทืืข ืื SNPs, ืึทืืื ืกืคึผืึทืจืง ืืื ืฆื ืขืคืขื ืขื ืึทืืข ืื ืืขืงืขืก ืฆื ืขืงืกืืจืึทืงื ืื ืืื ืคึฟืึธืจืืึทืฆืืข ืขืก ืืืจืฃ.
ืื ืืขืจืขืกืืื ืืื, Parquet ืก ืคืขืืืงืืึทื (ืืื ืจืขืงืึทืืขื ืืื) ืงืึทืืคึผืจืขืฉืึทื ืืืคึผ, ืกื ืึทืคึผื, ืืื ื ืืฉื ืกืคึผืืืืึทืืืข. ืืขืจืืืขืจ, ืืขืืขืจ ืขืงืกืึทืงืืืืขืจ ืืื ืืขืืืขื ืกืืึทืง ืืืืฃ ืื ืึทืจืืขื ืคืื ืึทื ืคึผืึทืงืื ื ืืื ืืึทืื ืืึธืืืื ื ืื ืคืื 3,5 ืืืืืืืื ืืึทืืึทืกืขื.
ืืื ืก ืคึฟืึทืจืฉืืืื ืื ืคึผืจืึธืืืขื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืกืึธืจืืื ื ืืื ืฉืืืขืจ, ืกืคึผืขืฆืืขื ืืืื ืื ืืึทืื ืืขื ืขื ืคืื ืื ืืขืจืืขืืืืื.
ืขืก ืืื ืืืจ ืืืืกืืขืืขื ืื ืืขืฆื ืืื ืืื ืคืืจืฉืืื ืขื ืืขื ืืืืช ืคืื ืืขื ืคืจืืืืขื. ืืื ื ืึธืจ ืืืจืฃ ืฆื ืกืึธืจื ืื ืืึทืื ืืืจื SNP ืืืึทื, ื ืืฉื ืืืจื ืืขื ืืฉื. ืืขืจื ืึธื ืขืืืขืืข SNPs ืืืขื ืืืื ืกืืึธืจื ืืื ืึท ืืึทืืื ืืขืจ ืืึทืื ืฉืืืง, ืืื ืืขืจ "ืงืืื" ืคึฟืื ืงืฆืืข ืคืื โโโโParquet "ืขืคืขื ืขื ืืืืื ืืืื ืื ืืืขืจื ืืื ืืื ืื ืงืืื" ืืืขื ืืืืึทืื ืืื ืืื ืึทืืข ืืืื ืืืื. ืฆืื ืืึทืืืืขืจื, ืกืึธืจืืื ื ืืืจื ืืืืืึทื ื ืคืื ืจืึธืื ืฆืขืืืึธืจืคื ืืืืขืจ ืึท ืงื ืืื ืคึผืจืืืื ืฆื ืืืื ืึท ืฉืืืขืจ ืึทืจืืขื.
ืืืจ ื ืขืืขื ืึทืืืขืจืืืึทืื ืงืืึทืก ืืื ืงืึธืืขืืข: "ืึท, ืงืืื ืืืื ืขืจ ืืืืืช ืืืขืื ืงืึทืืคึผืืืืืืฉืึทื ืึทื ืงืึทืืคึผืืขืงืกืืื ืคืื ืึทืืข ืื ืกืึธืจืืื ื ืึทืืืขืจืืืึทืื"
ืืื ืืจืืื ื ืฆื ืกืึธืจื ืืืืฃ ืึท ืืืึทื ืืื ืึท 20TB ืืืฉ: "ืคืืจืืืืก ื ืขืื ืืึธืก ืึทืืื ืืึทื ื?" ืจืื ืืืขื ืืฉื.
โ Nick Strayer (@NicholasStrayer)
AWS ืืืื ืืืฉืืืื ื ืืฉื ืึทืจืืืกืืขืื ืึท ืฆืืจืืงืฆืึธื ืืืืึทื ืคืื ืื ืกืืื "ืืื ืืื ืึท ืืืกืืจืึทืงืืึทื ืชึผืืืื". ื ืึธื ืืื ืืืืคื ืกืึธืจืืื ื ืืืืฃ Amazon Glue, ืขืก ืืื ืืขืืืคื ืคึฟืึทืจ 2 ืืขื ืืื ืงืจืึทืฉื.
ืืืึธืก ืืืขืื ืฆืขืืืืืื ื?
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืคึผืึทืจืืืฉืึทื ื ืืื ืกืคึผืึทืจืง ืืืื ืืืื ืืึทืืึทื ืกื.
ืืขืจื ืึธื ืืื ืืขืงืืืขื ืึทืจืืืฃ ืืื ืืขื ืืขืืึทื ืง ืคืื ืคึผืึทืจืืืฉืึทื ืื ื ืืึทืื ืืื ืืฉืจืึธืืึธืกืึธืืื. ืขืก ืืขื ืขื 23 ืคืื ืืื (ืืื ืขืืืขืืข ืืขืจ ืืืื ืืืจ ื ืขืืขื ืืื ืืฉืืื ืืืืึธืืฉืึธื ืืจืืึทื ืื ืึท ืืื ืึทื ืืึทืคึผื ืืงืืืืช).
ืืึธืก ืืืขื ืืึธืื ืืืจ ืฆื ืฉืคึผืึทืืื ืื ืืึทืื ืืื ืงืืขื ืขืจืขืจ ืฉืืืงืขืจ. ืืืื ืืืจ ืืืืื ืืืืื ืืืื ืฉืืจื ืฆื ืื ืกืคึผืึทืจืง ืขืงืกืคึผืึธืจื ืคึฟืื ืงืฆืืข ืืื ืื ืงืืื ืฉืจืืคื partition_by = "chr", ืืขืืึธืื ืื ืืึทืื ืืึธื ืืืื ืฆืขืืืืื ืืื ืืึทืงืึทืฅ.
ืื ืืขื ืึธืืข ืืืฉืืืื ืคืื ืกื ืคืจืึทืืืึทื ืฅ ืืขืจืืคื ืืฉืจืึธืืึธืกืึธืืขืก.
ืฆืื ืืึทืืืืขืจื, ืขืก ืืื ื ืืฉื ืึทืจืืขืื. ืืฉืจืึธืืึธืกืึธืืขืก ืืึธืื ืคืึทืจืฉืืืขื ืข ืกืืืขืก, ืืืึธืก ืืืื ืคืึทืจืฉืืืขื ืข ืึทืืึทืื ืฅ ืคืื ืืื ืคึฟืึธืจืืึทืฆืืข. ืืืก ืืืื ื ืื ืื ืืืืคืืืื ืืืืก ืกืคืืจืง ืืื ืืขืฉืืงื ืฆื ืืจืืขืืขืจ ืืขื ืขื ื ืืฉื ืืืืื ืกืืจื ืืื ืืื ืืืื ืคืืจืขื ืืืื, ืืืืื ืืืื ื ืึธืืขืก ืืืื ืืื ืคืืจืขื ืืืงื ืคืจื ืืื ืืขื ืขื ืืขืืืขื ืืืืืืง. ื ื ืืืืคืืืืข ื ืืฒื ืข ื ืืืข ืจ ืคืืจืขื ืืืง ื ืืขืฐืืจื . ืืืขืจ ืืืขื ืืขื ืืึธื ืืขืืขืื ืคึฟืึทืจ ืืืื SNP, ืื ืืืืึทืืึทื ืก ืืืืืขืจ ืืขืคึฟืืจื ืคึผืจืึธืืืขืืก. ืื ืคึผืจืืึทื ืคืื ืคึผืจืึทืกืขืกืื ื ืกื ืคึผืก ืืืืฃ ืืจืขืกืขืจืข ืืฉืจืึธืืึธืกืึธืืขืก (ืืึธืก ืืื, ืืื ืืืจ ืืืืื ืฆื ืืึทืงืืืขื ืืึทืื) ืืื ืืืืื ืืืงืจืืกื ืืื ืืืขืื ืึท ืคืึทืงืืึธืจ ืคืื 10. ืึท ืกื, ืึธืืขืจ ื ืืฉื ืืขื ืื.
ืืืึธืก ืืืื ืืืจ ืืืืื ืขืก ืืื ืืคืืื ืงืืขื ืขืจืขืจ ืืืืื?
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืงืืื ืืึธื ืคึผืจืืืืจื ืฆื ืืึธื 2,5 ืืืืืึธื ืคึผืึทืจืืืฉืึทื ื ืืื ืึทืืข.
ืืื ืืึทืฉืืึธืกื ืฆื ืืืื ืึทืืข ืืืืก ืืื ืฆืขืืืืื ืืขืืขืจ SNP. ืื ื ืื ื ืื ื ืคืืจืืืืขืจื , ื ื ื ื ืคืืจืืฒืข ื ืืฒื ืข ื ืืขืฐืข ื ืคื ื ืืืฒื ืข ืคืืจืืื . ืขืก ืืื ืืขืืืขื ืึท ืฉืืขืื ืืขืืึทื ืง. ืืื ืืขืืืืื ื ืงืืื ืืื ืฆืืืขืืขืื ืึทื ืืืืฉืืืืืง ืฉืืจื partition_by = 'snp'. ืื ืึทืจืืขื ืื ืืขืืืืื ืืื ืื ืืขืืืืื ืฆื ืืืกืคืืจื. ื ืืึธื ืฉืคึผืขืืขืจ ืืื ืึธืคึผืืขืฉืืขืื ืืื ืืขืืขื ืึทื ืขืก ืืื ื ืึธื ืืึธืจื ืืฉื ืืขืฉืจืืื ืฆื ืก 3, ืึทืืื ืืื ืืขืืจืืขื ืื ืึทืจืืขื. ืขืก ืงืืงื ืืื ืงืืื ืืื ืืขืฉืจืืื ืื ืืขืจืืืืืื ืืขืงืขืก ืฆื ืึท ืคืึทืจืืึธืจืื ืึธืจื ืืื S3, ืึท ืคึผืืึทืฅ ืคืื ืืขืงืขืก, ืืึธืืขืจ ืึท ืคึผืึธืจ ืคืื ืืืืืึธื. ืืขืจ ืจืขืืืืืึทื, ืืืื ืืจืืึทื ืงืึธืกื ืืขืจ ืืื ืึท ืืืืื ื ืืึธืืืึทืจืก ืืื ืืื ื ืืฉื ืืืืข ืืืื ืืึทืืจืขื.
ืคึผืึทืจืืืืืึธื + ืกืึธืจืืื ื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืกืึธืจืืื ื ืืื ื ืึธื ืฉืืืขืจ, ืืื ืืื ืื ื ืกืคึผืึทืจืง.
ืืืึทื ืืขืฆืืข ืคึผืจืืืื ืฆื ืฆืขืืืืื ืื ืืืึทืืืื ืืืจ ืฆืขืืืืื ืื ืืฉืจืึธืืึธืกืึธืืื ืืื ืืขืจื ืึธื ืกืึธืจืืื ื ืืขืืขืจ ืฆืขืืืืืื ื. ืืื ืืขืึธืจืืข, ืืึธืก ืืืึธืื ืคืึทืจืืืืขืจื ืืขืืขืจ ืึธื ืคึฟืจืขื ืืืืึทื ืื ืืขืืขืื SNP ืืึทืื ืืึธืื ืฆื ืืืื ืื ืึท ืืืกื ืคึผืึทืจืงืื ืืฉืึทื ืืงืก ืืื ืึท ืืขืืขืื ืงืืื. ืฆืื ืืึทืืืืขืจื, ืกืึธืจืืื ื ืืคืืื ืคึผืึทืจืืืฉืึทื ื ืืึทืื ืืื ืืขืืืขื ืึท ืฉืืืขืจ ืึทืจืืขื. ืืื ืึท ืจืขืืืืืึทื, ืืื ืกืืืืืฉื ืฆื EMR ืคึฟืึทืจ ืึท ืื ืื ืงื ืืื ืืื ืืขืืืืื ื ืึทืื ืฉืืึทืจืง ืื ืกืืึทื ืกืื (C5.4xl) ืืื Sparklyr ืฆื ืฉืึทืคึฟื ืึท ืืขืจ ืคืืขืงืกืึทืืึทื ืืืึธืจืงืคืืึธืื ...
# Sparklyr snippet to partition by chr and sort w/in partition
# Join the raw data with the snp bins
raw_data
group_by(chr) %>%
arrange(Position) %>%
Spark_write_Parquet(
path = DUMP_LOC,
mode = 'overwrite',
partition_by = c('chr')
)... ืึธืืขืจ, ืื ืึทืจืืขื ืืื ื ืึธื ื ืืฉื ืืขืขื ืืืงื. ืืื ืงืึทื ืคืืืืขืจื ืขืก ืืื ืคืึทืจืฉืืืขื ืข ืืืขืื: ืืขืืืืงืกื ืื ืืืงืึธืจื ืึทืืึทืงืืืฉืึทื ืคึฟืึทืจ ืืขืืขืจ ืึธื ืคึฟืจืขื ืขืงืกืึทืงืืืืขืจ, ืืขืืืืื ื ื ืึธืืื ืืื ืึท ืืจืืืก ืกืืืข ืคืื โโโโืืืงืึธืจื, ืืขืืืืื ื ืืจืึธืืงืึทืกื ืืืขืจืืึทืืึทืื (ืืจืึธืืงืึทืกื ืืืขืจืืึทืืึทืื), ืึธืืขืจ ืืขืืขืจ ืืึธื ืืึธืก ืืื ืืขืืืขื ืืึทืื-ืืืืืขื, ืืื ืืืกืืขืืืืืึทื ืื ืขืงืกืึทืงืืืืขืจื ืื ืืขืืืืื ืฆื ืคืึทืจืืึธืื ืืื ืึทืืฅ ืคืืจืฉืืืคื.
ืืขืจืืืึทื ืืืงื: ืึทืืื ืขืก ืืืืื.
โ Nick Strayer (@NicholasStrayer)
ืืื ืืื ืฉืืื ืืขืจ ืฉืขืคืขืจืืฉ
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืืื ืกืคึผืขืฆืืขื ืืึทืื ืจืืงืืืืืขืจื ืกืคึผืขืฆืืขื ืกืึทืืืฉืึทื ื.
ืืขืืขืจ SNP ืืื ืึท ืฉืืขืืข ืืืขืจื. ืืึธืก ืืื ืึท ื ืืืขืจ ืงืึธืจืึทืกืคึผืึทื ืืื ื ืฆื ืื ื ืืืขืจ ืคืื ืืึทืกืขืก ืฆืืืืืขื ืืืึทื ืืจืึธืืึธืกืึธื. ืืึธืก ืืื ืึท ืฉืืื ืืื ื ืึทืืืจืืขื ืืืขื ืฆื ืึธืจืืึทื ืืืืจื ืืื ืืืขืจ ืืึทืื. ืืื ืขืจืฉืืขืจ ืืื ืืขืืืืื ืฆื ืฆืขืืืืื ืืืจื ืืงืืืืช ืคืื ืืขืืขืจ ืืจืึธืืึธืกืึธื. ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืฉืืขืืขืก 1 - 2000, 2001 - 4000, ืืื"ื ื. ืึธืืขืจ ืื ืคึผืจืึธืืืขื ืืื ืึทื SNPs ืืขื ืขื ื ืืฉื ืืืืึทื ืื ืคืื ืื ืืขืจืืขืืืืื ืืืืขืจ ืื ืืฉืจืึธืืึธืกืึธืืื, ืึทืืื ืื ืืจืืคึผืข ืกืืืขืก ืืืขื ืืืื ืืืืขืจ ืึทื ืืขืจืฉ.
ืืื ืึท ืจืขืืืืืึทื, ืืื ืืขืงืืืขื ืฆื ืึท ืืจืืืงืืึทืื ืคืื ืฉืืขืืขืก ืืื ืงืึทืืขืืึธืจืืขืก (ืจืึทื ื). ื ืืฆื ืื ืฉืืื ืืึทืื ืืึธืืืื ืืึทืื, ืืื ืืขืืืคื ืึท ืืงืฉื ืฆื ืืึทืงืืืขื ืึท ืจืฉืืื ืคืื ืืื ืฆืืง ืกื ืคึผืก, ืืืืขืจ ืฉืืขืืขืก ืืื ืืฉืจืึธืืึธืกืึธืืื. ืืขืจื ืึธื ืืื ืกืึธืจืืืจื ืื ืืึทืื ืืื ืืขืืขืจ ืืจืึธืืึธืกืึธื ืืื ืืขืืืืื ืกื ืคึผืก ืืื ืืจืืคึผืขืก (ืืื) ืคืื ืึท ืืขืืขืื ืืจืืืก. ืืื ืก ืืึธืื 1000 ืกื ืคึผืก ืืขืืขืจ. ืืึธืก ืืึธื ืืืจ ืืขืืขืื ืื SNP-ืฆื-ืืจืืคึผืข-ืคึผืขืจ-ืืฉืจืึธืืึธืกืึธื ืฉืืืืืช.
ืืื ืื ืกืืฃ, ืืื ืืขืืืื ืืจืืคึผืขืก (ืืื) ืคืื 75 ืกื ืคึผืก, ืื ืกืืื ืืืขื ืืืื ืืขืจืงืืขืจื ืืื ืื.
snp_to_bin <- unique_snps %>%
group_by(chr) %>%
arrange(position) %>%
mutate(
rank = 1:n()
bin = floor(rank/snps_per_bin)
) %>%
ungroup()ืขืจืฉืืขืจ ืคึผืจืืืืจื ืืื Spark
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืกืคึผืึทืจืง ืึทืืืจืขืืึทืืืึธื ืืื ืฉื ืขื, ืึธืืขืจ ืคึผืึทืจืืืฉืึทื ืื ื ืืื ื ืึธื ืืืึทืขืจ.
ืืื ืืขืืืืื ืฆื ืืืืขื ืขื ืืขื ืงืืืื (2,5 ืืืืืึธื ืจืึธืื) ืืึทืื ืจืึทื ืืื ืกืคึผืึทืจืง, ืคืึทืจืืื ืื ืขืก ืืื ืื ืจืื ืืึทืื ืืื ืฆืขืืืืื ืขืก ืืืจื ืื ื ืื ืฆืืืขืืืืื ืืืึทื. bin.
# Join the raw data with the snp bins
data_w_bin <- raw_data %>%
left_join(sdf_broadcast(snp_to_bin), by ='snp_name') %>%
group_by(chr_bin) %>%
arrange(Position) %>%
Spark_write_Parquet(
path = DUMP_LOC,
mode = 'overwrite',
partition_by = c('chr_bin')
)
ืืื ืืื ืืขื ืืฆื sdf_broadcast(), ืึทืืื ืกืคึผืึทืจืง ืืืืืกื ืึทื ืขืก ืืึธื ืฉืืงื ืื ืืึทืื ืจืึทื ืฆื ืึทืืข ื ืึธืืื. ืืึธืก ืืื ื ืืฆืืง ืืืื ืื ืืึทืื ืืขื ืขื ืงืืืื ืืื ืืจืืืก ืืื ืคืืจืืื ืื ืคึฟืึทืจ ืึทืืข ืืึทืกืงืก. ืึทื ืืขืจืฉ, ืกืคึผืึทืจืง ืคืจืืืื ืฆื ืืืื ืงืืื ืืื ืืืกืืจืืืืืฅ ืืึทืื ืืื ืืืจืฃ, ืืืึธืก ืงืขื ืคืึทืจืฉืึทืคื ืกืืึธืืืึทืื ื.
ืื ื ืฐืืืข ืจ ืื ื ืืฒ ื ืืืืขืข ื ื ืืฉ ื ืืขืืจืืข ื : ื ื ืืืืคืืืืข ื ืืื ื ืืขืืจืืข ื ื ืฆืฒื , ืคืืจืขื ืืืง ื ืืข ื ืคืืจืืฒ ื ืื ื ืื ื , ืฐ ื ื ื ืขืงืืขืงืืืืจ ื ืืื ื ืื ื ืื ืืขืืืื ื ืื ื ืฆืขืืืืืื ื , ืืื ื ื ืฒ ืื ืืขืืืื ื ืืืจืืคืืื .
ืึทืืื ื AWK
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืื ืืืืกื ื ืืฉื ืฉืืึธืคื ืืืขื ืืืจ ืืขื ื ืืขืืขืจื ื ืื ืืึทืกืืงืก. ืฉืืจืื ืขืืขืฆืขืจ ืฉืืื ืกืึทืืืื ืืืื ืคึผืจืึธืืืขื ืฆืืจืืง ืืื ืื 1980 ืก.
ืืื ืฆื ืืขื ืคืื ื, ืื ืกืืื ืคึฟืึทืจ ืึทืืข ืืืื ืคืืืืืขืจื ืืื ืกืคึผืึทืจืง ืืื ืืขืืืขื ืื ืฆืขืืืฉืื ื ืคืื ืืึทืื ืืื ืืขื ืงื ืืื. ืืึธืืขืจ ืื ืกืืืืึทืฆืืข ืงืขื ืขื ืืืื ืืืคึผืจืืืื ืืื ืคืึทืจ-ืืึทืืึทื ืืืื ื. ืืื ืืึทืฉืืึธืกื ืฆื ืคึผืจืืืืจื ืกืคึผืืืืื ื ืื ืจืื ืืขืงืกื ืืึทืื ืืื ืฉืคืืืื ืคืื ืืฉืจืึธืืึธืกืึธืืื, ืึทืืื ืืื ืืึธืืคึผื ืฆื ืฆืืฉืืขืื ืกืคึผืึทืจืง ืืื "ืคืึทืจ-ืืืืื" ืืึทืื.
ืืื ืืขืืืื ืืืืฃ StackOverflow ืคึฟืึทืจ ืืื ืฆื ืฉืคึผืึทืืื ืืืจื ืืืึทื ืืืึทืืืขืก ืืื ืืขืคึฟืื ืขื ืืื AWK ืืืจ ืงืขื ืขื ืฉืคึผืึทืืื ืึท ืืขืงืกื ืืขืงืข ืืืจื ืืืึทื ืืืึทืืืขืก ืืืจื ืฉืจืืืื ืขืก ืืื ืึท ืฉืจืืคื ืืื ืืื ืฆื ืฉืืงื ืื ืจืขืืืืืึทืื ืฆื stdout.
ืืื ืืขืฉืจืืื ืึท ืืึทืฉ ืฉืจืืคื ืฆื ืคึผืจืืืืจื ืขืก. ืืึทืื ืืึธืืืื ืืืื ืขืจ ืคืื ืื ืคึผืึทืงืืืืฉื TSVs ืืื ืืึทื ืึทื ืคึผืึทืงื ืขืก ื ืืฆื gzip ืืื ืืขืฉืืงื ืฆื awk.
gzip -dc path/to/chunk/file.gz |
awk -F 't'
'{print $1",..."$30">"chunked/"$chr"_chr"$15".csv"}'ืขืก ืืขืืจืืขื!
ืคืืืื ื ืื ืงืึธืจืขืก
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: gnu parallel - ืขืก ืืื ืึท ืืึทืืืฉ ืืึทื, ืึทืืขืืขื ืืึธื ื ืืฆื ืขืก.
ืื ืฆืขืฉืืืืื ื ืืื ืืขืืืขื ืืึทื ืฅ ืคึผืึทืืขืืขื ืืื ืืืขื ืืื ืกืืึทืจืืขื htopืฆื ืงืึธื ืืจืึธืืืจื ืื ื ืืฆื ืคืื ืึท ืฉืืึทืจืง (ืืื ืืืึทืขืจ) EC2 ืืืึทืฉืคึผืื, ืขืก ืคืืจืงืขืจื ืืืืก ืึทื ืืื ืืขืืืืื ื ืืืืื ืืืื ืืึทืจืฅ ืืื ืืืขืื 200 ืืขืืืืืืื ืคืื ืืึผืจืื. ืฆื ืกืึธืืืืข ืื ืคึผืจืึธืืืขื ืืื ื ืืฉื ืคืึทืจืืืจื ืึท ืคึผืืึทืฅ ืคืื ืืขืื, ืืืจ ืืึธืื ืฆื ืจืขืืขื ืขื ืืืืก ืืื ืฆื ืคึผืึทืจืึทืืขืืืืืจื ืื ืึทืจืืขื. ืฆืื ืืืืง, ืืื ืึท ืืขืืึทืืจืข ืึทืืืืืื ื ืืื ืืื ืืึธื ืืขืคึฟืื ืขื ืึท ืงืึทืคึผืืื ืคึฟืื ืืืฉืขืจืึธื ืืึทื ืกืกืขื ืก ืืืขืื ืคึผืึทืจืึทืืขืืืืึทืฆืืข. ืคืื ืขืก ืืื ืืขืืขืจื ื ืืืขืื gnu parallel, ืึท ืืืืขืจ ืคืืขืงืกืึทืืึทื ืืืคึฟื ืคึฟืึทืจ ืืืคึผืืึทืืขื ืื ื ืืืืืืืืจืขืึทืืื ื ืืื ืืื ืืงืก.
ืืืขื ืืื ืกืืึทืจืืขื ืื ืคึผืึทืจืืืฉืึทื ืื ื ืืื ืื ื ืืึทืข ืคึผืจืึธืฆืขืก, ืึทืืฅ ืืื ืืขืืืขื ืืื, ืึธืืขืจ ืขืก ืืื ืืขืืืขื ื ืึธื ืึท ืืึทืืึทืื ืขืง - ืืึทืื ืืึธืืืื ื S3 ืึทืืืืฉืขืงืฅ ืฆื ืืืกืง ืืื ื ืืฉื ืืืืขืจ ืฉื ืขื ืืื ื ืืฉื ืืึธืจ ืคึผืึทืจืึทืืขืืืืขื. ืฆื ืคืึทืจืจืืืื ืืขื, ืืื ืืึธื ืืึธืก:
- ืืื ืืขืคืื ืขื ืึทื ืขืก ืืื ืืขืืืขื ืฆื ืื ืกืืจืืืขื ื ืื S3 ืืจืืคืงืืคืืข ืืื ืข ืืืืึทื ืืื ืื ืจืขืจื - ืืื ืืข, ืืึธืจ ืืืืืึทื ืืืืื ื ืื ืืขืจืืืืืื ืกืืึธืจืืืืฉ ืืืืฃ ืืืกืง. ืืขื ืืืื ืืื ืงืขื ืขื ืืืกืืืืื ืฉืจืืืื ืจืื ืืึทืื ืฆื ืืืกืง ืืื ื ืืฆื ืืคืืื ืงืืขื ืขืจืขืจ, โโโโืืื ืืขืจืืืขืจ ืืฉืืคึผืขืจ, ืกืืึธืจืืืืฉ ืืืืฃ AWS.
- ืืึทื ืฉืึทืคึฟื
aws configure set default.s3.max_concurrent_requests 50ืืืืขืจ ืืขืืืืงืกื ืื ื ืืืขืจ ืคืื ืคึฟืขืืขื ืืืึธืก AWS CLI ื ืืฆื (ืืืจื ืคืขืืืงืืึทื ืขืก ืืขื ืขื 10). - ืืื ืกืืืืืฉื ืฆื ืึทื EC2 ืืืึทืฉืคึผืื ืึธืคึผืืืืืืขื ืคึฟืึทืจ ื ืขืฅ ืืืืงืืึทื, ืืื ืื ืืจืืื n ืืื ืืขื ื ืึธืืขื. ืืื ืืึธืื ืืขืคึฟืื ืขื ืึทื ืื ืึธื ืืืขืจ ืคืื ืคึผืจืึทืกืขืกืื ื ืืึทืื ืืืขื ื ืืฆื n-ืื ืกืืึทื ืกืขืก ืืื ืืขืจ ืืื ืงืึทืืคึผืึทื ืกืืืืึทื ืืืจื ืื ืคืึทืจืืจืขืกืขืจื ืืื ืืึธืืืื ื ืืืืงืืึทื. ืคึฟืึทืจ ืจืืึฟ ืืึทืกืงืก ืืื ืืขืืืืื ื c5n.4xl.
- ืืขืืืื
gzipืืืืฃ , ืืึธืก ืืื ืึท ืืืืคึผ ืืขืฆืืึทื ืืืึธืก ืงืขื ืขื ืืึธื ืงืื ืืื ืื ืฆื ืคึผืึทืจืึทืืขืืืืืจื ืื ืืืืืขืก ื ืื-ืคึผืขืจืึทืืขืืืืื ืึทืจืืขื ืคืื ืืืงืึทืืคึผืจืขืกืื ื ืืขืงืขืก (ืืึธืก ืืขืืึธืืคึฟื ืื ืืื ืืกืืขืจ).
# Let S3 use as many threads as it wants
aws configure set default.s3.max_concurrent_requests 50
for chunk_file in $(aws s3 ls $DATA_LOC | awk '{print $4}' | grep 'chr'$DESIRED_CHR'.csv') ; do
aws s3 cp s3://$batch_loc$chunk_file - |
pigz -dc |
parallel --block 100M --pipe
"awk -F 't' '{print $1",..."$30">"chunked/{#}_chr"$15".csv"}'"
# Combine all the parallel process chunks to single files
ls chunked/ |
cut -d '_' -f 2 |
sort -u |
parallel 'cat chunked/*_{} | sort -k5 -n -S 80% -t, | aws s3 cp - '$s3_dest'/batch_'$batch_num'_{}'
# Clean up intermediate data
rm chunked/*
doneืื ืกืืขืคึผืก ืืขื ืขื ืงืึทืืืืื ื ืืื ืืขืืขืจ ืื ืืขืจืข ืฆื ืืึทืื ืึทืืฅ ืึทืจืืขื ืืืืขืจ ืืขืฉืืืื ื. ืืืจื ืื ืงืจืืกืื ื ืืจืืคืงืืคืืข ืกืคึผืืื ืืื ืืืืืึทื ืืืืื ื ืืืกืง ืฉืจืืืื, ืืื ืงืขื ืืืฆื ืคึผืจืึธืฆืขืก ืึท 5 ืืขืจืึทืืืืข ืคึผืขืงื ืืื ืืืืื ืึท ืืืกื ืฉืขื.
ืขืก ืืื ืืึธืจื ืืฉื ืืืกืขืจ ืืื ืฆื ืืขื ืึทืืข ืื ืงืึธืจืขืก ืืืึธืก ืืืจ ืืึทืฆืึธืื ืคึฟืึทืจ AWS. ืืึทื ืง ืฆื gnu-parallel ืืื ืงืขื ืขื ืึทื ืืืคึผ ืืื ืฉืคึผืึทืืื ืึท 19 ืืื ืงืกืื ืคึผืื ืงื ืืื ืฉื ืขื ืืื ืืื ืงืขื ืขื ืืจืืคืงืืคืืข ืขืก. ืืื ืงืขื ื ืืฉื ืืคืืื ืืึทืงืืืขื ืึท ืึธื ืฆืื ืื ืฆื ืืืืคื ืืขื.
โ Nick Strayer (@NicholasStrayer)
ืืขืจ ืืืืขืขื ืืึธื ืืึธืื ืืขืจืืื ื 'TSV'. ืืืื.
ื ืืฆื ื ืื ืคึผืึทืจืกืขื ืืึทืื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืกืคึผืึทืจืง ืืืืงืก ืื ืงืึธืืคึผืจืขืกืกืขื ืืึทืื ืืื ืืื ื ืืฉื ืืื ืงืึทืืืืื ืื ื ืคึผืึทืจืืืฉืึทื ื.
ืืืฆื ืื ืืึทืื ืืขื ืขื ืืขืืืขื ืืื S3 ืืื ืึท ืึทื ืคึผืึทืงื (ืืืืขื ืขื: ืฉืขืจื) ืืื ืืึทืื-ืึธืจืืขืจื ืคึฟืึธืจืืึทื, ืืื ืืื ืงืขื ืฆืืจืืงืงืืืขื ืฆื ืกืคึผืึทืจืง ืืืืืขืจ. ื ืืืขืจืจืึทืฉื ืึทืืืืืืึทื ืืืจ: ืืื ืืืืืขืจ ื ืื ืึทื ืืขืจืฉ ืฆื ืืขืจืืจืืืื ืืืึธืก ืืื ืืขืืืืื! ืขืก ืืื ืืขืืืขื ืืืืขืจ ืฉืืืขืจ ืฆื ืืึธืื ืกืคึผืึทืจืง ืคึผืื ืงื ืืื ืื ืืึทืื ืืขื ืขื ืคึผืึทืจืืืฉืึทื ื. ืืื ืืคืืื ืืืขื ืืื ืืื ืืืก ืืขืืื, ืืื ืืื ืืจืืืกืืขืฉืืขืื ืื ืขืก ืืขื ืขื ืืขืืืขื ืฆืืคืื ืืืืฆืืช (95 ืืืืื ื), ืืื ืืืขื ืืื ืืื ืืขื ืืฆื coalesce ืจืืืืกื ืืืืขืจ ื ืืืขืจ ืฆื ืืืืึทื ืืืืึทืฅ, ืืึธืก ืืจืืึฟ ืืืื ืฆืขืืืืืื ื. ืืื ืืื ืืืืขืจ ืึทื ืืึธืก ืงืขื ืืืื ืคืึทืจืคืขืกืืืงื, ืึธืืขืจ ื ืึธื ืึท ืคึผืึธืจ ืคืื ืืขื ืคืื ืืืื ืืื ืงืขื ื ืืฉื ืืขืคึฟืื ืขื ืึท ืืืืืื ื. ืืื ืืืืขื ืืฉืึทืืืึทืื ืคืึทืจืืืง ืึทืืข ืื ืืึทืกืงืก ืืื ืกืคึผืึทืจืง, ืืึธืืฉ ืขืก ืืขื ืืืขื ืึท ืืฉืขืช ืืื ืืืื ืฉืคึผืึทืืื ืคึผืึทืจืงืืืขื ืืขืงืขืก ืืขื ืขื ื ืืฉื ืืืืขืจ ืงืืืื (~ 200 ืงื). ืึธืืขืจ, ืื ืืึทืื ืืื ืืขืืืขื ืืื ืขืก ืืื ืืืจืฃ.
ืืืื ืงืืืื ืืื ืึทื ืืืืึทื, ืืืื ืืขืจืืขื!
ืืขืกืืื ื ืืืืข ืกืคึผืึทืจืง ืคึฟืจืืื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืกืคึผืึทืจืง ืืื ืฆื ืคืื ืึธืืืืขืจืืขื ืืืขื ืกืึทืืืืื ื ืคึผืฉืื ืคึผืจืึธืืืขืืก.
ืืืจื ืืึทืื ืืึธืืืื ื ืื ืืึทืื ืืื ืึท ืงืืื ืคึฟืึธืจืืึทื, ืืื ืืื ืืขืืืขื ืืืืืืช ืฆื ืคึผืจืืืืจื ืื ืืืืงืืึทื. ืฉืืขืื ืึท R ืฉืจืืคื ืฆื ืืืืคื ืึท ืืืืข ืกืคึผืึทืจืง ืกืขืจืืืขืจ, ืืื ืืขืืึธืื ืืึธืืืื ืึท ืกืคึผืึทืจืง ืืึทืื ืจืึทื ืคึฟืื ืื ืกืคึผืขืกืืคืืขื ืคึผืึทืจืงืืืขื ืืจืืคึผืข ืกืืึธืจืืืืฉ (ืืื). ืืื ืืขืคืจืืืื ืฆื ืืึธืื ืึทืืข ืื ืืึทืื ืึธืืขืจ ืงืขื ื ืืฉื ืืึทืงืืืขื ืกืคึผืึทืจืงืืืจ ืฆื ืืขืจืงืขื ืขื ืื ืคึผืึทืจืืืฉืึทื ืื ื.
sc <- Spark_connect(master = "local")
desired_snp <- 'rs34771739'
# Start a timer
start_time <- Sys.time()
# Load the desired bin into Spark
intensity_data <- sc %>%
Spark_read_Parquet(
name = 'intensity_data',
path = get_snp_location(desired_snp),
memory = FALSE )
# Subset bin to snp and then collect to local
test_subset <- intensity_data %>%
filter(SNP_Name == desired_snp) %>%
collect()
print(Sys.time() - start_time)ืื ืืืจืืคืืจืื ื ืืขื ืืืขื 29,415 ืกืขืงืื ืืขืก. ืคืื ืืขืกืขืจ, ืึธืืขืจ ื ืืฉื ืฆื ืืื ืคึฟืึทืจ ืืึทืกืข ืืขืกืืื ื ืคืื ืขืคึผืขืก. ืืื ืืขืจืฆื, ืืื ืงืขื ื ืืฉื ืคืึทืจืืืืขืจื ืื ืืื ืื ืืื ืงืึทืืฉืื ื ืืืืึทื ืืืขื ืืื ืืขืคืจืืืื ืฆื ืงืึทืฉ ืึท ืืึทืื ืจืึทื ืืื ืืึผืจืื, ืกืคึผืึทืจืง ืฉืืขื ืืืง ืงืจืึทืฉื, ืืคืืื ืืืขื ืืื ืึทืืึทืงืืืืื ืืขืจ ืืื 50 ืืืืืืืื ืคืื ืืึผืจืื ืฆื ืึท ืืึทืืึทืกืขื ืืืึธืก ืืืืื ืืืืื ืืงืขืจ ืืื 15.
ืฆืืจืืงืงืืืขื ืฆื AWK
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืึทืกืกืึธืกืืึทืืืืืข ืขืจืืื ืืื AWK ืืขื ืขื ืืืืขืจ ืขืคืขืงืืืื.
ืืื ืืืื ืืขืืขื ืึทื ืืื ืงืขื ืืขืจืืจืืืื ืืขืืขืจ ืกืคึผืืื. ืืื ืืขืืขื ืง ืึทื ืืื ืึท ืืืื ืืขืจืืขื ืืื ืืืืขื ืขื ืืืขืื ืึท ืงืื ืฉืืจืื ืืขืจืืคื "" ืืกืขื ืฉืึทืื, ืืึธืก ืืขื ืขื ืฉืืืกื-ืืืขืจื ืคึผืขืจื, ืืืึธืก ืคึฟืึทืจ ืขืืืขืืข ืกืืื ืืขื ืขื ืืขืจืืคื ืืืคืขืจืขื ืืื ืืื AWK, ืืื ืืขืจืืืขืจ ืืื ืขืคืขืก ืืื ื ืืฉื ืืจืึทืืื ืคืื ืืืขืื ืืื. ืจืืงืึธืื ืึทื ืืขืจ ืืขืจืืื "ืึทืกืกืึธืกืืึทืืืืืข ืขืจืืื" ืืื ืคืื ืขืืืขืจ ืืื ืืขืจ ืืขืจืืื "ืฉืืืกื-ืืืขืจื ืคึผืึธืจ". ืืคืืื ืืืื ืืืจ , ืืืจ ืืืขื ื ืืฉื ืืขื ืืขื ืืขืจืืื ืืึธืจื, ืึธืืขืจ ืืืจ ืืืขื ืืขืคึฟืื ืขื ืึทืกืกืึธืกืืึทืืืืืข ืขืจืืื! ืืื ืืขืจืฆื, ืื "ืฉืืืกื-ืืืขืจื ืคึผืึธืจ" ืืื ืืขืจืกื ืึธืคื ืคืืจืืื ืื ืืื ืืึทืืึทืืืืกืื, ืึทืืื ืขืก ืืื ืคืื ืืขืจ ืืื ืขื ืฆื ืคืึทืจืืืืึทืื ืขืก ืืื ืึท ืืึทืฉืืึทืคึผ. ืืื ืืืื ืืขืืขื ืึทื ืืื ืงืขื ื ืืฆื ืื ืึทืกืกืึธืกืืึทืืืืืข ืขืจืืื ืฆื ืคืึทืจืืื ืื ืืืื SNPs ืืื ืึท ืืื ืืืฉ ืืื ืจืื ืืึทืื ืึธื ื ืืฆื ืกืคึผืึทืจืง.
ืฆื ืืึธื ืืึธืก, ืืื ืื AWK ืฉืจืืคื ืืื ืืขืืืืื ื ืืขื ืืืึธืง BEGIN. ืืึธืก ืืื ืึท ืฉืืืง ืคืื ืงืึธื ืืืึธืก ืืื ืขืงืกืึทืงืืืืึทื ืืืืืขืจ ืืขืจ ืขืจืฉืืขืจ ืฉืืจื ืคืื ืืึทืื ืืื ืืืจืืืขืืื ืืขื ืฆื ืื ืืืืคึผื ืืืฃ ืคืื ืื ืฉืจืืคื.
join_data.awk
BEGIN {
FS=",";
batch_num=substr(chunk,7,1);
chunk_id=substr(chunk,15,2);
while(getline < "snp_to_bin.csv") {bin[$1] = $2}
}
{
print $0 > "chunked/chr_"chr"_bin_"bin[$1]"_"batch_num"_"chunk_id".csv"
}
ืงืึธืืขืงืืืื while(getline...) ืืึธืืืื ืึทืืข ืจืึธืื ืคืื ืื CSV ืืจืืคึผืข (ืืื), ืฉืืขืื ืื ืขืจืฉืืขืจ ืืืึทื (SNP ื ืึธืืขื) ืืื ืืขืจ ืฉืืืกื ืคึฟืึทืจ ืื ืึทืกืกืึธืกืืึทืืืืืข ืืขื ืืข bin ืืื ืืขืจ ืฆืืืืืืขืจ ืืืขืจื (ืืจืืคืข) ืืืก ืืืขืจื. ืืขืจื ืึธื ืืื ืื ืืืึธืง { }, ืืืึธืก ืืื ืขืงืกืึทืงืืืืึทื ืืืืฃ ืึทืืข ืฉืืจืืช ืคืื ืื ืืืืคึผื ืืขืงืข, ืืขืืขืจ ืฉืืจื ืืื ืืขืฉืืงื ืฆื ืืขืจ ืจืขืืืืืึทื ืืขืงืข, ืืืึธืก ื ืขืื ืึท ืืื ืฆืืง ื ืึธืืขื ืืืคึผืขื ืืื ื ืืืืฃ ืืืึทื ืืจืืคึผืข (ืืื): ..._bin_"bin[$1]"_....
ืืืขืจืืึทืืึทืื batch_num ะธ chunk_id ืืึทืืฉื ืื ืืึทืื ืฆืืืขืฉืืขืื ืืืจื ืื ืจืขืจื - ืืื ืืข, ืึทืืืืืืื ืึท ืจืึทืกืข ืฆืืฉืืึทื ื ืืื ืืขืืขืจ ืืืจืืคืืจืื ื ืคืึธืืขื ืคืืืกื ืืืง parallel, ืืขืฉืจืืื ืฆื ืืืื ืืืืืขื ืข ืืื ืฆืืง ืืขืงืข.
ืืื ื ืืื ืฆืขืืืึธืจืคื ืึทืืข ืื ืจืื ืืึทืื ืืื ืคืึธืืืขืจืก ืืืืฃ ืืฉืจืึธืืึธืกืึธืืื ืืื ืงืก ืืืืขืจ ืคืื ืืืื ืคืจืืขืจืืืงื ืขืงืกืคึผืขืจืืืขื ื ืืื AWK, ืืืฆื ืืื ืงืขื ืฉืจืืึทืื ืื ืื ืืขืจ ืืึทืฉ ืฉืจืืคื ืฆื ืคึผืจืึธืฆืขืก ืืืื ืืจืึธืืึธืกืึธื ืืื ืึท ืฆืืื ืืื ืฉืืงื ืืืคึผืขืจ ืคึผืึทืจืืืฉืึทื ื ืืึทืื ืฆื S3.
DESIRED_CHR='13'
# Download chromosome data from s3 and split into bins
aws s3 ls $DATA_LOC |
awk '{print $4}' |
grep 'chr'$DESIRED_CHR'.csv' |
parallel "echo 'reading {}'; aws s3 cp "$DATA_LOC"{} - | awk -v chr=""$DESIRED_CHR"" -v chunk="{}" -f split_on_chr_bin.awk"
# Combine all the parallel process chunks to single files and upload to rds using R
ls chunked/ |
cut -d '_' -f 4 |
sort -u |
parallel "echo 'zipping bin {}'; cat chunked/*_bin_{}_*.csv | ./upload_as_rds.R '$S3_DEST'/chr_'$DESIRED_CHR'_bin_{}.rds"
rm chunked/*
ืืขืจ ืฉืจืืคื ืืื ืฆืืืื ืกืขืงืฉืึทื ื parallel.
ืืื ืืขืจ ืขืจืฉืืขืจ ืึธืคึผืืืืืื ื, ืืึทืื ืืขื ืขื ืืืืขื ืขื ืคืื ืึทืืข ืืขืงืขืก ืืื ืืื ืคึฟืึธืจืืึทืฆืืข ืืืืฃ ืื ืืขืืขืื ืืจืึธืืึธืกืึธื, ืืื ืื ืืึทืื ืืขื ืขื ืคืื ืื ืืขืจืืขืืืืื ืืืืขืจ ืคึฟืขืืขื, ืืืึธืก ืคืึทืจืฉืคึผืจืืืื ืื ืืขืงืขืก ืืื ืื ืฆืื ืขืืขื ืืจืืคึผืขืก (ืืื). ืฆื ืืืกืืืืื ืจืึทืกืข ืื ืึธืื ืืืขื ืงืืืคื ืคึฟืขืืขื ืฉืจืืึทืื ืฆื ืืขืจ ืืขืืืืงืขืจ ืืขืงืข, AWK ืคึผืึทืกืื ืื ืืขืงืข ื ืขืืขื ืฆื ืฉืจืืึทืื ืืึทืื ืฆื ืคืึทืจืฉืืืขื ืข ืขืจืืขืจ, ืืืฉื. chr_10_bin_52_batch_2_aa.csv. ืืื ืึท ืจืขืืืืืึทื, ืคืืืข ืงืืืื ืืขืงืขืก ืืขื ืขื ืืืฉืืคื ืืืืฃ ืื ืืืกืง (ืคึฟืึทืจ ืืขื ืืื ืืขืืืืื ื ืืขืจืึทืืืืข EBS ืืืึทืืืืื).
ืงืึทื ืืืืืขืจ ืคืื ืื ืจืืข ืึธืคึผืืืืืื ื parallel ืืืื ืืืจื ืื ืืจืืคึผืขืก (ืืื) ืืื ืงืึทืืืืื ื ืืืืขืจ ืืืื ืืขืงืขืก ืืื ืคึผืจืึธืกื CSV c catืืื ืืขืืึธืื ืกืขื ืื ืืื ืคึฟืึทืจ ืึทืจืืืกืคืืจื.
ืืจืึธืืงืึทืกืืื ื ืืื ืจ?
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืืืจ ืงืขื ืขื ืงืึธื ืืึทืงื stdin ะธ stdout ืคืื ืึท R ืฉืจืืคื, ืืื ืืขืจืืืขืจ ื ืืฆื ืขืก ืืื ืื ืจืขืจื - ืืื ืืข.
ืืืจ ืงืขื ืืึธืื ืืืืขืจืงื ืื ืฉืืจื ืืื ืืืื Bash ืฉืจืืคื: ...cat chunked/*_bin_{}_*.csv | ./upload_as_rds.R.... ืขืก ืืจืึทื ืกืืืืฅ ืึทืืข ืงืึทื ืงืึทืืึทื ืืืืึทื ืืจืืคึผืข ืืขืงืขืก (ืืื) ืืื ืื R ืฉืจืืคื ืืื ืื. {} ืืื ืึท ืกืคึผืขืฆืืขื ืืขืื ืืง parallel, ืืืึธืก ืื ืกืขืจืฅ ืงืืื ืืึทืื ืขืก ืกืขื ืื ืฆื ืื ืกืคึผืขืกืืคืืขื ืืืึทื ืืืืึทื ืืื ืื ืืึทืคึฟืขื ืืื. ืึธืคึผืฆืืข {#} ืืื ืึท ืืื ืฆืืง ืคืึธืืขื ืฉืืึทื, ืืื {%} ืจืขืคึผืจืึทืืขื ืฅ ืื ืึทืจืืขื ืฉืคึผืขืืื ื ืืืขืจ (ืจืืคึผืืืื, ืึธืืขืจ ืงืืื ืืึธื ืกืืืืึทืืืืื ืืึทืกืื). ื ืจืฉืืื ืคืื ืึทืืข ืึธืคึผืฆืืขืก ืงืขื ืขื ืืืื ืืขืคึฟืื ืขื ืืื
#!/usr/bin/env Rscript
library(readr)
library(aws.s3)
# Read first command line argument
data_destination <- commandArgs(trailingOnly = TRUE)[1]
data_cols <- list(SNP_Name = 'c', ...)
s3saveRDS(
read_csv(
file("stdin"),
col_names = names(data_cols),
col_types = data_cols
),
object = data_destination
)
ืืืขื ืึท ืืืึทืืขืืืืืง file("stdin") ืืจืึทื ืกืืืืืขื ืฆื readr::read_csv, ืื ืืึทืื ืืืืขืจืืขืืขืฆื ืืื ืื R ืฉืจืืคื ืืื ืืึธืืืื ืืื ืึท ืจืึทื, ืืืึธืก ืืื ืืขืืึธืื ืืื ืื ืคืึธืจืขื .rds-ืืขืงืข ื ืืฆื aws.s3 ืืขืฉืจืืื ืืืืึทื ืฆื S3.
RDS ืืื ืขืคึผืขืก ืืื ืึท ืืื ืืขืจ ืืืขืจืกืืข ืคืื โโืคึผืึทืจืงืืืขื, ืึธื ืื ืคืจืืื ืคืื ืจืขืื ืขืจ ืกืืึธืจืืืืฉ.
ื ืึธืื ืคืึทืจืขื ืืืงื ืืขื ืืึทืฉ ืฉืจืืคื ืืึธื ืืื ืืึทืงืืืขื ืึท ืคึผืขืงื .rds-ืคืืืขืก ืืืื ืืื S3, ืืืึธืก ืขืจืืืืื ืืืจ ืฆื ื ืืฆื ืขืคืขืงืืืื ืงืึทืืคึผืจืขืฉืึทื ืืื ืืขืืืื-ืืื ืืืืคึผืก.
ืืจืึธืฅ ืื ื ืืฆื ืคืื ืืึธืจืืึธื ืจ, ืึทืืฅ ืืขืืจืืขื ืืืืขืจ ืืขืฉืืืื ื. ื ืื ืกืึทืคึผืจืืืืื ืืื, ืื ืืืืื ืคืื R ืืืึธืก ืืืืขื ืขื ืืื ืฉืจืืึทืื ืืึทืื ืืขื ืขื ืืขืืกื ืึธืคึผืืืืืืขื. ื ืึธื ืืขืกืืื ื ืืืืฃ ืืืื ืืืื-ืกืืืื ืืจืึธืืึธืกืึธื, ืื ืึทืจืืขื ืืื ืืขืขื ืืืงื ืืืืฃ ืึท C5n.4xl ืืืึทืฉืคึผืื ืืื ืืืขืื ืฆืืืื ืฉืขื.
S3 ืืืืืืืืฉืึทื ื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืืึทื ืง ืฆื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืงืืื ืืืขื, S3 ืงืขื ืขื ืฉืขืคึผื ืคืืืข ืืขืงืขืก.
ืืื ืืื ืืขืืืขื ืืึทืืึธืจืื ืฆื S3 ืืืึธืื ืืืื ืืืืืืช ืฆื ืฉืขืคึผื ืื ืคืืืข ืืขืงืขืก ืืืึธืก ืืขื ืขื ืืจืึทื ืกืคืขืจื ืฆื ืขืก. ืืื ืงืขื ืืึทืื ืืื ืขื ืื ืืขืงืข ื ืขืืขื, ืึธืืขืจ ืืื ืืืึธืื S3 ืงืืงื ืคึฟืึทืจ ืืื?
ืคืึธืืืขืจืก ืืื S3 ืืขื ืขื ื ืึธืจ ืคึฟืึทืจ ืืืืึทืื, ืืื ืคืึทืงื, ืื ืกืืกืืขื ืืื ื ืืฉื ืืื ืืขืจืขืกืืจื ืืื ืืขื ืกืืืืึธื /.
ืขืก ืืืืก ืึทื S3 ืจืขืคึผืจืึทืืขื ืฅ ืืขืจ ืืจื ืฆื ืึท ืืึทืืื ืืขืจ ืืขืงืข ืืื ืึท ืคึผืฉืื ืฉืืืกื ืืื ืึท ืกืึธืจื ืคืื ืืึทืฉ ืืืฉ ืึธืืขืจ ืืึธืงืืืขื ื-ืืืืืจื ืืึทืืึทืืืืก. ื ืขืืขืจ ืงืขื ืขื ืืืื ืงืึทื ืกืืืขืจื ืืื ืึท ืืืฉ, ืืื ืืขืงืขืก ืงืขื ืขื ืืืื ืืขืืืืื ืจืขืงืึธืจืืก ืืื ืืขื ืืืฉ.
ืืื ื ืืืืงืืึทื ืืื ืขืคืขืงืืืืืงืืึทื ืืขื ืขื ืืืืืืืง ืฆื ืืึทืื ืึท ื ืืฅ ืืื ืึทืืึทืืึธื, ืขืก ืก ืงืืื ืืืขืจืจืึทืฉื ืึทื ืืขื ืฉืืืกื-ืืื-ืึท-ืืขืงืข-ืืจื ืกืืกืืขื ืืื ืคืจืืงืื ื ืึธืคึผืืืืืืขื. ืืื ืืึธื ืืขืคึผืจืึผืืื ืืขืคึฟืื ืขื ืึท ืืึทืืึทื ืก: ืึผืื ืืื ืืึธื ื ืืฉื ืืึธืื ืฆื ืืึทืื ืึท ืกื ืืงืฉืืช, ื ืึธืจ ืึทื ืื ืืงืฉืืช ืืขื ืขื ืืขืฉืืืื ื ืืืจืืืขืคึฟืืจื ืืขืืืึธืจื. ืขืก ืคืืจืงืขืจื ืืืืก ืึทื ืขืก ืืื ืืขืกืืขืจ ืฆื ืืึทืื ืืืขืื 20 ืืื ืืขืงืขืก. ืืื ืืจืึทืืื ืืืื ืืืจ ืคืึธืจืืขืฆื ืฆื ืึทืคึผืืึทืืืื, ืืืจ ืงืขื ืขื ืืขืจืืจืืืื ืึท ืคืึทืจืืจืขืกืขืจื ืืื ืืืืงืืึทื (ืืืฉื, ืืึทืื ืึท ืกืคึผืขืฆืืขื ืขืืขืจ ื ืึธืจ ืคึฟืึทืจ ืืึทืื, ืึทืืื ืจืืืืกืื ื ืื ืืจืืืก ืคืื ืื ืืืงืึทืคึผ ืืืฉ). ืืืขืจ ืขืก ืืื ืืขืืืขื ืงืืื ืฆืืื ืึธืืขืจ ืืขืื ืคึฟืึทืจ ืืืืึทืืขืจ ืืงืกืคึผืขืจืึทืืึทื ืฅ.
ืืืึธืก ืืืขืื ืงืจืืึทื ืงืึทืืคึผืึทืืึทืืืืึทืื?
ืืืึธืก ืืื ืืขืืขืจื ื: ืื ื ืืืขืจ ืืืื ืกืืื ืคืื ืืืืืกืืึทื ืฆืืื ืืื ืฆื ืคืจื ืึธืคึผืืืืืืื ื ืืืื ืกืืึธืจืืืืฉ ืืืคึฟื.
ืืื ืืขื ืคืื ื, ืขืก ืืื ืืืืขืจ ืืืืืืืง ืฆื ืคืจืขืื ืืื: "ืคืืจืืืืก ื ืืฆื ืึท ืคึผืจืึทืคึผืจืืืึทืืขืจื ืืขืงืข ืคึฟืึธืจืืึทื?" ืื ืกืืื ืืืื ืืื ืืึธืืืื ื ืืืืงืืึทื (ืืืืคึผืคึผืขื ืงืกืื ืืขืงืขืก ืืขื ืืืขื 7 ืืื ืืขืจ ืฆื ืืึทืกืข) ืืื ืงืึทืืคึผืึทืืึทืืืืึทืื ืืื ืืื ืืืขืจ ืืืึธืจืงืคืืึธืื. ืืื ืงืขื ืืืขืจืงืืขืจื ืืืื ืจ ืงืขื ืขื ืืืืื ืืึธืื ืคึผืึทืจืงืืืขื (ืึธืืขืจ ืขืจืึธื) ืืขืงืขืก ืึธื ืื ืกืคึผืึทืจืง ืืึทืกืข. ืึทืืขืืขื ืืื ืืื ืืืขืจ ืืึทื ื ืืฆื ืจ, ืืื ืืืื ืืื ืืึทืจืคึฟื ืฆื ืืขืจ ืื ืืึทืื ืฆื ืื ืื ืืขืจ ืคึฟืึธืจืืึทื, ืืื ื ืึธื ืืึธืื ืื ืึธืจืืืื ืขื ืืขืงืกื ืืึทืื, ืึทืืื ืืื ืงืขื ืขื ื ืึธืจ ืืืืคื ืื ืจืขืจื - ืืื ืืข ืืืืืขืจ.
ืึธืคึผืืืื ืคืื ืึทืจืืขื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืื ืืืืกื ื ืืฉื ืคึผืจืืืืจื ืฆื ืึทืคึผืืึทืืืื ืืืฉืึธืืก ืืึทื ืืืึทืื, ืืึธืื ืื ืงืึธืืคึผืืืืขืจ ืืึธื ืืึธืก.
ืืื ืืึธืื ืืืืึทืื ืื ืืืึธืจืงืคืืึธืื ืืืืฃ ืืืื ืืจืึธืืึธืกืึธื, ืืืฆื ืืื ืืึทืจืคึฟื ืฆื ืคึผืจืึธืฆืขืก ืึทืืข ืื ืื ืืขืจืข ืืึทืื.
ืืื ืืขืืืืื ืฆื ืืึทืคึผื ืขืืืขืืข EC2 ืื ืกืืึทื ืกืื ืคึฟืึทืจ ืงืึทื ืืืขืจืืฉืึทื, ืึธืืขืจ ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื ืืื ืืื ืืขืืืขื ืืขืจืฉืจืึธืงื ืฆื ืืึทืงืืืขื ืึท ืืืืขืจ ืึทื ืืึทืืึทื ืกื ืืึทืกืข ืืืืขืจ ืคืึทืจืฉืืืขื ืข ืคึผืจืึทืกืขืกืื ื ืืืฉืึธืืก (ืคึผืื ืงื ืืื ืกืคึผืึทืจืง ืืขืืืื ืคืื ืึทื ืืึทืืึทื ืกื ืคึผืึทืจืืืฉืึทื ื). ืืื ืึทืืืฉืึทื, ืืื ืืื ื ืืฉื ืืื ืืขืจืขืกืืจื ืืื ืจืืืืื ื ืืืื ืืืึทืฉืคึผืื ืคึผืขืจ ืืจืึธืืึธืกืึธื, ืืืืึทื ืคึฟืึทืจ AWS ืึทืงืึทืื ืฅ ืขืก ืืื ืึท ืคืขืืืงืืึทื ืืืืื ืคืื 10 ืื ืกืืึทื ืกืื.
ืืขืจื ืึธื ืืื ืืึทืฉืืึธืกื ืฆื ืฉืจืืึทืื ืึท ืฉืจืืคื ืืื ืจ ืฆื ืึทืคึผืืึทืืืื ืคึผืจืึทืกืขืกืื ื ืืืฉืึธืืก.
ืขืจืฉืืขืจ, ืืื ืืขืืขืื S3 ืฆื ืจืขืืขื ืขื ืืื ืคืื ืกืืึธืจืืืืฉ ืคึผืืึทืฅ ืืขืืขืจ ืืจืึธืืึธืกืึธื ืคืึทืจื ืืืขื.
library(aws.s3)
library(tidyverse)
chr_sizes <- get_bucket_df(
bucket = '...', prefix = '...', max = Inf
) %>%
mutate(Size = as.numeric(Size)) %>%
filter(Size != 0) %>%
mutate(
# Extract chromosome from the file name
chr = str_extract(Key, 'chr.{1,4}.csv') %>%
str_remove_all('chr|.csv')
) %>%
group_by(chr) %>%
summarise(total_size = sum(Size)/1e+9) # Divide to get value in GB
# A tibble: 27 x 2
chr total_size
<chr> <dbl>
1 0 163.
2 1 967.
3 10 541.
4 11 611.
5 12 542.
6 13 364.
7 14 375.
8 15 372.
9 16 434.
10 17 443.
# โฆ with 17 more rows
ืืขืจื ืึธื ืืื ืืขืฉืจืืื ืึท ืคึฟืื ืงืฆืืข ืืืึธืก ื ืขืื ืื ืืึทื ืฅ ืืจืืืก, ืฉืึทืคืึทืื ืื ืกืืจ ืคืื ืื ืืฉืจืึธืืึธืกืึธืืื, ืฆืขืืืืื ืืื ืืื ืืจืืคึผืขืก num_jobs ืืื ืืขืจืฆืืืื ืืืจ ืืื ืึทื ืืขืจืฉ ืื ืกืืืขืก ืคืื ืึทืืข ืคึผืจืึทืกืขืกืื ื ืืืฉืึธืืก ืืขื ืขื.
num_jobs <- 7
# How big would each job be if perfectly split?
job_size <- sum(chr_sizes$total_size)/7
shuffle_job <- function(i){
chr_sizes %>%
sample_frac() %>%
mutate(
cum_size = cumsum(total_size),
job_num = ceiling(cum_size/job_size)
) %>%
group_by(job_num) %>%
summarise(
job_chrs = paste(chr, collapse = ','),
total_job_size = sum(total_size)
) %>%
mutate(sd = sd(total_job_size)) %>%
nest(-sd)
}
shuffle_job(1)
# A tibble: 1 x 2
sd data
<dbl> <list>
1 153. <tibble [7 ร 3]>ืืขืจื ืึธื ืืื ืืขืืืคื ืืืจื ืึท ืืืืื ื ืฉืึทืคืึทืื ื ืืฆื ืคึผืืจืจ ืืื ืืืืกืืขืจืืืืืื ืืขืจ ืืขืกืืขืจ.
1:1000 %>%
map_df(shuffle_job) %>%
filter(sd == min(sd)) %>%
pull(data) %>%
pluck(1)
ืึทืืื ืืื ืขื ืืืงื ืืื ืืื ืึท ืืึทื ื ืคืื ืืึทืกืงืก ืืืึธืก ืืขื ืขื ืืขืืืขื ืืืืขืจ ืขื ืืขื ืืื ืืจืืืก. ืื ื ืื ื ื ื ืจ ืืขืืืื ื ืื ื ื ืืจืืืืข ืจ ืฉืืืืฃ , ืฐื ืก ืื ื ืืขืืืื ื ืื ื ื ืืจืืืืข ืจ ืฉืืืืฃ for. ืืขื ืึทืคึผืืึทืืึทืืืืฉืึทื ืืขื ืืืขื ืืืขืื 10 ืืื ืื ืฆื ืฉืจืืึทืื. ืืื ืืึธืก ืืื ืคืื ืืืืื ืืงืขืจ ืืื ืืื ืืืึธืื ืคืึทืจืืจืขื ืืขื ืืืืฃ ืืึทื ืืืึทืื ืงืจืืืืืื ื ืืึทืกืงืก ืืืื ืืื ืืขื ืขื ืึทื ืืึทืืึทื ืกื. ืืขืจืืืขืจ, ืืื ืืจืึทืืื ืึทื ืืื ืืื ืืขืืืขื ืจืขืื ืืื ืืขื ืคึผืจืืืืืึทื ืขืจื ืึทืคึผืืึทืืึทืืืืฉืึทื.
for DESIRED_CHR in "16" "9" "7" "21" "MT"
do
# Code for processing a single chromosome
fiืืื ืื ืกืืฃ ืืื ืืืืื ืื ืฉืึทืืืึทืื ืืึทืคึฟืขื:
sudo shutdown -h now
... ืืื ืึทืืฅ ืืขืืจืืขื ืืืืก! ื ืืฆื ืื AWS CLI, ืืื ืืืืคืฉืืืื ืื ืกืืึทื ืกืื ื ืืฆื ืื ืึธืคึผืฆืืข user_data ืืื ืืื ืืขืืขืื ืืืฉืฉืจืืคืื ืคืื ืืืืขืจืข ืืืืคืืืื ืคืืจื ืคืืจืืจืืขืื. ืืื ืืขืืืคื ืืื ืคืึทืจืืึทืื ืืื ืืืืืึธืืึทืืืฉ, ืึทืืื ืืื ืืื ื ืืฉื ืืึทืฆืึธืื ืคึฟืึทืจ ืขืงืกืืจืข ืคึผืจืึทืกืขืกืื ื ืืึทืื.
aws ec2 run-instances ...
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=<<job_name>>}]"
--user-data file://<<job_script_loc>>ืืื ืก ืคึผืึทืงื!
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืื ืึทืคึผื ืืึธื ืืืื ืคึผืฉืื ืคึฟืึทืจ ืื ืื ืืื ืืืืืืงืืึทื ืคืื ื ืืฆื.
ืืขืกืึธืฃ ืืื ืืึทื ืื ืืึทืื ืืื ืื ืจืขืื ืึธืจื ืืื ืคืึธืจืขื. ืขืก ืืื ื ืึธืจ ืืขืืืืื ืฆื ืคืึทืจืคึผืึธืฉืขืืขืจื ืืขื ืคึผืจืึธืฆืขืก ืคืื ื ืืฆื ืืึทืื ืืื ืคืื ืืื ืืขืืืขื ืฆื ืืึทืื ืขืก ืืจืื ืืขืจ ืคึฟืึทืจ ืืืื ืืืจืื. ืืื ืืขืืืืื ืฆื ืืึทืื ืึท ืคึผืฉืื ืึทืคึผื ืคึฟืึทืจ ืงืจืืืืืื ื ืจืืงืืืขืก. ืืืื ืืื ืืขืจ ืฆืืงืื ืคึฟื ืืื ืืึทืฉืืืกื ืฆื ืืึทืฉืืืืขื ืคืื .rds ืฆื ืคึผืึทืจืงืืืขื ืืขืงืขืก, ืืึธืก ืืึธื ืืืื ืึท ืคึผืจืึธืืืขื ืคึฟืึทืจ ืืืจ, ื ืืฉื ืคึฟืึทืจ ืืืื ืืืจืื. ืคึฟืึทืจ ืืขื ืืื ืืึทืฉืืึธืกื ืฆื ืืึทืื ืึท ืื ืขืจืืขื ืจ ืคึผืขืงื.
ืืืืขื ืืื ืืึธืงืืืขื ื ืึท ืืืืขืจ ืคึผืฉืื ืคึผืขืงื ืืื ืืืืื ืึท ืืืกื ืืึทืื ืึทืงืกืขืก ืคืึทื ืืงืฉืึทื ื ืึธืจืืึทื ืืืืจื ืึทืจืื ืึท ืคึฟืื ืงืฆืืข get_snp. ืืื ืืืื ืืขืืืื ืึท ืืืขืืืืืื ืคึฟืึทืจ ืืืื ืืืจืื , ืึทืืื ืืื ืงืขื ืขื ืืืืื ืืขื ืืืืฉืคืืื ืืื ืืึทืงืืืืขื ืืืืฉืึทื.
ืกืืึทืจื ืงืึทืืฉืื ื
ืืืึธืก ืืึธื ืืื ืืขืืขืจื ื: ืืืื ืืืื ืืึทืื ืืขื ืขื ืืขืืื ื ืฆืืืขืืจืืื, ืงืึทืืฉืื ื ืืืขื ืืืื ืืจืื ื!
ืืื ื ืืืื ืขืจ ืคืื ืื ืืืืคึผื ืืืึธืจืงืคืืึธืื ืืขืืืขื ืื ืืขืจ ืืขืืืืงืขืจ ืึทื ืึทืืืกืืก ืืึธืืขื ืฆื ืื SNP ืคึผืขืงื, ืืื ืืึทืฉืืึธืกื ืฆื ื ืืฆื ืืื ืื ื ืฆื ืืืื ืืืึทืืข. ืืืขื ืืจืึทื ืกืืืืื ื ืืึทืื ืืืจื SNP, ืึทืืข ืืื ืคึฟืึธืจืืึทืฆืืข ืคืื โโืืขืจ ืืจืืคึผืข (ืืื) ืืื ืึทืืึทืืฉื ืฆื ืื ืืืืืขืงืขืจื ืืืืคืขืฅ. ืึทื ืืื, ืึทืื ืงืืืืจืื ืงืขื ืขื (ืืื ืืขืึธืจืืข) ืคืึทืจืืืืขืจื ืื ืคึผืจืึทืกืขืกืื ื ืคืื ื ืืึทืข ืคึฟืจืืื.
# Part of get_snp()
...
# Test if our current snp data has the desired snp.
already_have_snp <- desired_snp %in% prev_snp_results$snps_in_bin
if(!already_have_snp){
# Grab info on the bin of the desired snp
snp_results <- get_snp_bin(desired_snp)
# Download the snp's bin data
snp_results$bin_data <- aws.s3::s3readRDS(object = snp_results$data_loc)
} else {
# The previous snp data contained the right bin so just use it
snp_results <- prev_snp_results
}
...
ืืืขื ืืื ืืืืขื ืืขื ืคึผืขืงื, ืืื ืืขืืืคื ืคืืืข ืืขื ืืฉืืึทืจืงืก ืฆื ืคืึทืจืืืืึทืื ืืืืงืืึทื ืืืขื ืืื ื ืืฆื ืคืึทืจืฉืืืขื ืข ืืขืืืึธืืก. ืืื ืจืขืงืึธืืขื ืืืจื ื ืืฉื ืฆื ืคืึทืจืืึธืื ืืขื, ืืืืึทื ืืื ืื ืจืขืืืืืึทืื ืืขื ืขื ืืืืืขืจืืื. ืืืฉื, dplyr::filter ืืื ืืขืืืขื ืคืื ืคืึทืกืืขืจ ืืื ืงืึทืคึผืืฉืขืจืื ื ืจืึธืื ื ืืฆื ืื ืืขืงืกืื ื-ืืืืืจื ืคึฟืืืืจืืจืื ื, ืืื ืจืืืจืืืืื ื ืึท ืืืื ืืืึทื ืคึฟืื ืึท ืืขืคืืืืขืจื ืืึทืื ืจืึทื ืืื ืคืื ืคืึทืกืืขืจ ืืื ื ืืฆื ืื ืืขืงืกืื ื ืกืื ืืึทืงืก.
ืืืืข ืืึธื ืึทื ืื ืืืืคืขืฅ prev_snp_results ืึผืืื ืื ืฉืืืกื snps_in_bin. ืืึธืก ืืื ืึท ืืขื ืืข ืคืื โโโโืึทืืข ืืื ืฆืืง SNPs ืืื ืึท ืืจืืคึผืข (ืืื), ืึทืืึทืืื ื ืืืจ ืฆื ืืขืฉืืืื ื ืงืึธื ืืจืึธืืืจื ืืืื ืืืจ ืฉืืื ืืึธืื ืืึทืื ืคืื ืึท ืคืจืืขืจืืืงื ืึธื ืคึฟืจืขื. ืขืก ืืืื ืืืื ืขืก ืืจืื ื ืฆื ืฉืืืืฃ ืืืจื ืึทืืข ืื SNPs ืืื ืึท ืืจืืคึผืข (ืืื) ืืื ืืขื ืงืึธื:
# Get bin-mates
snps_in_bin <- my_snp_results$snps_in_bin
for(current_snp in snps_in_bin){
my_snp_results <- get_snp(current_snp, my_snp_results)
# Do something with results
}ืจืขืืืืืึทืื
ืืืฆื ืืืจ ืงืขื ืขื (ืืื ืืึธืื ืื ืืขืืืืื ืฆื ืขืืขืก) ืืืืคื ืืึธืืขืืก ืืื ืกืื ืขืจืืึธืื ืืืึธืก ืืขื ืขื ืคืจืืขืจ ืื ืึทืงืกืขืกืึทืืึทื ืคึฟืึทืจ ืืื ืื. ืืขืจ ืืขืกืืขืจ ืืึทื ืืื ืึทื ืืืื ืืึทื ืืืจืื ืืึธื ื ืื ืืึธืื ืฆื ืืจืึทืืื ืืืขืื ืงืืื ืงืึทืืคึผืืึทืงืืืฉืึทื ื. ืืื ื ืึธืจ ืืึธืื ืึท ืคึฟืื ืงืฆืืข ืืืึธืก ืึทืจืืขื.
ืืื ืืึธืืฉ ืืขืจ ืคึผืขืงื ืกืคึผืขืจ ืืื ืื ืืขืืึทืืืก, ืืื ืืขืคืจืืืื ืฆื ืืึทืื ืื ืืึทืื ืคึฟืึธืจืืึทื ืคึผืฉืื ืืขื ืื ืึทื ืืื ืงืขื ืืขืคึฟืื ืขื ืขืก ืืืื ืืื ืคึผืืืฆืืื ื ืคืึทืจืฉืืืื ืื ืืึธืจืื ...
ืื ืืืืงืืึทื ืืื ืืขืืืืงืกื ืืืืขืจืงื. ืืืจ ืืืืฉืึทืืืึทืื ืืืขืจืงืืงื ืคืึทื ืืงืฉืึทื ืึทืื ืืึทืืืึทืืืง ืืขื ืึธืืข ืคืจืึทืืืึทื ืฅ. ืืื ืึทืืขืจ, ืืืจ ืงืขื ื ืืฉื ืืึธื ืืึธืก (ืขืก ืืื ืืขืืืขื ืฆื ืืืึทืขืจ), ืึธืืขืจ ืืืฆื, ืืึทื ืง ืฆื ืื ืืจืืคึผืข (ืืื) ืกืืจืืงืืืจ ืืื ืงืึทืืฉืื ื, ืึท ืืงืฉื ืคึฟืึทืจ ืืืื SNP ื ืขืื ืืื ืืืจืืฉื ืืืืขื ืืืืื ืืงืขืจ ืืื 0,1 ืกืขืงืื ืืขืก, ืืื ืื ืืึทืื ืืึทื ืืฅ ืืื ืึทืืื. ื ืืืขืจืืง ืึทื ืื ืงืึธืก ืคึฟืึทืจ ืก 3 ืืขื ืขื ืคึผืื ืึทืฅ.
ืืขืฆืื ืก, ืืื ืืึทื ืึท ืืืืฉื ืคืื ืจืึทื ืืืื ื 25+ ืื ืคืื ืจืื ืืขื ืึธืืืคึผืื ื ืืึทืื ืคึฟืึทืจ ืืืื ืืึทืืึธืจืึทืืึธืจืืข. ืืืขื ืืื ืกืืึทืจืืขื, ื ืืฆื ืึธื ืฆืื ืื ืืขื ืืืขื 8 ืืื ืื ืืื ืงืึธืกืื $ 20 ืฆื ืึธื ืคึฟืจืขื ืึท SNP. ื ืึธื ื ืืฆื AWK + ืฆื ืคึผืจืึธืฆืขืก, ืขืก ืืืฆื ื ืขืื ืืืืื ืืงืขืจ ืืื ืึท 10 ืกืขืงืื ืืข ืืื ืงืึธืก $0.00001. ืืืึทื ืคึผืขืจืืขื ืืขื ืืขืืืื ืขื.
โ Nick Strayer (@NicholasStrayer)
ืกืึธืฃ
ืืขืจ ืึทืจืืืงื ืืื ืืืื ื ืืฉื ืึท ืคืืจืขืจ. ืื ืืืืืื ื ืืื ืืขืืืขื ืืืื, ืืื ืึผืืขื ืืืืขืจ ื ืืฉื ืึธืคึผืืืืึทื. ืืื, ืขืก ืืื ืึท ืืจืึทืืืึทืืื ื. ืืื ืืืืื ืื ืืขืจืข ืฆื ืคึฟืึทืจืฉืืืื ืึทื ืึทืืึท ืืืกืืืฉืึทื ื ืืขื ืขื ื ืืฉื ืืึธืจ ืืขืฉืืคื ืืื ืื ืงืึธืคึผ, ืืื ืืขื ืขื ืืขืจ ืจืขืืืืืึทื ืคืื ืคึผืจืึธืฆืขืก ืืื ืืขืืช. ืืืื, ืืืื ืืืจ ืืืื ืคึฟืึทืจ ืึท ืืึทืื ืืขืืขืจื ืืขืจ, ืืึทืืื ืืื ืืืื ืื ื ืึทื ืื ื ืืฆื ืคืื ืื ืืืฉืืจืื ืืคืขืงืืืืืื ืจืืงืืืืืขืจื ืืขืจืคืึทืจืื ื ืืื ืืขืจืคืึทืจืื ื ืงืึธืก ืืขืื. ืืื ืืื ืฆืืคืจืืื ืึทื ืืื ืืขืืื ืื ืืืื ืฆื ืืึทืฆืึธืื, ืึธืืขืจ ืคืืืข ืื ืืขืจืข ืืืืก ืงืขื ืขื ืืึธื ืื ืืขืืืข ืึทืจืืขื ืืขืกืขืจ ืืื ืืืจ ืืืขืื ืงืืื ืืึธื ืืึธืื ืื ืืขืืขืื ืืืื ืืืืึทื ืคืื ืืึทื ืื ืคืื ืืขืื ืฆื ืคึผืจืืืืจื ืืคืืื.
ืืจืืืก ืืึทืื ืืืฉืืจืื ืืขื ืขื ืืืขืจืกืึทืืึทื. ืืืื ืืืจ ืืึธื ืื ืฆืืื, ืืืจ ืงืขื ืขื ืึผืืขื ืืืืขืจ ืฉืจืืึทืื ืึท ืคืึทืกืืขืจ ืืืืืื ื ืืื ืงืืื ืืึทืื ืจืืื ืืงืื ื, ืกืืึธืจืืืืฉ ืืื ืืงืกืืจืึทืงืฉืึทื ืืขืงื ืืงืก. ืืขืกืึธืฃ ืขืก ืงืืื ืึทืจืึธืคึผ ืฆื ืึท ืคึผืจืืึทื-ื ืืฅ ืึทื ืึทืืืกืืก.
ืืืึธืก ืืื ืืขืืขืจื ื:
- ืขืก ืืื ืงืืื ืืืืืง ืืืขื ืฆื ืคึผืึทืจืกืืจื 25 ืื ืืื ืึท ืฆืืึทื;
- ืืืื ืึธืคึผืืขืืื ืืื ืื ืืจืืืก ืคืื ืืืื ืคึผืึทืจืงืืืขื ืืขืงืขืก ืืื ืืืืขืจ ืึธืจืืึทื ืืืึทืฆืืข;
- ืคึผืึทืจืืืฉืึทื ื ืืื ืกืคึผืึทืจืง ืืืื ืืืื ืืึทืืึทื ืกื;
- ืืื ืึทืืืขืืืื, ืงืืื ืืึธื ืคึผืจืืืืจื ืฆื ืืึทืื 2,5 ืืืืืึธื ืคึผืึทืจืืืฉืึทื ื;
- ืกืึธืจืืื ื ืืื ื ืึธื ืฉืืืขืจ, ืืื ืืืื ืืึทืฉืืขืืืงื ืกืคึผืึทืจืง;
- ืืื ืกืคึผืขืฆืืขื ืืึทืื ืจืืงืืืืืขืจื ืกืคึผืขืฆืืขื ืกืึทืืืฉืึทื ื;
- ืึธื ืฆืื ืื ืึทืืืจืขืืึทืืืึธื ืืื ืฉื ืขื, ืึธืืขืจ ืคึผืึทืจืืืฉืึทื ืื ื ืืื ื ืึธื ืืืึทืขืจ;
- ืฉืืึธืฃ ื ืืฉื ืืืขื ืืขื ืืขืจื ื ืืืจ ืื ืืึทืกืืงืก, ืขืืขืฆืขืจ ืืื ืืืกืืึธืืข ืฉืืื ืกืึทืืืื ืืืื ืคึผืจืึธืืืขื ืฆืืจืืง ืืื ืื 1980 ืก;
gnu parallel- ืืึธืก ืืื ืึท ืืึทืืืฉ ืืึทื, ืึทืืขืืขื ืืึธื ื ืืฆื ืขืก;- ืกืคึผืึทืจืง ืืืืงืก ืื ืงืึธืืคึผืจืขืกืกืขื ืืึทืื ืืื ืืื ื ืืฉื ืืื ืงืึทืืืืื ืื ื ืคึผืึทืจืืืฉืึทื ื;
- ืกืคึผืึทืจืง ืืื ืฆื ืคืื ืึธืืืืขืจืืขื ืืืขื ืกืึทืืืืื ื ืคึผืฉืื ืคึผืจืึธืืืขืืก;
- AWK ืก ืึทืกืกืึธืกืืึทืืืืืข ืขืจืืื ืืขื ืขื ืืืืขืจ ืขืคืขืงืืืื;
- ืืืจ ืงืขื ืขื ืงืึธื ืืึทืงื
stdinะธstdoutืคืื ืึท R ืฉืจืืคื, ืืื ืืขืจืืืขืจ ื ืืฆื ืขืก ืืื ืื ืจืขืจื - ืืื ืืข; - ืืึทื ืง ืฆื ืกืืึทืจื ืืจื ืืืคึผืืึทืืขื ืืืืฉืึทื, S3 ืงืขื ืขื ืคึผืจืึธืฆืขืก ืคืืืข ืืขืงืขืก;
- ืื ืืืืคึผื ืกืืื ืคึฟืึทืจ ืืืืกื ืฆืืื ืืื ืคึผืจืืืึทืืฉืืจืื ืึธืคึผืืืืืืื ื ืืืื ืกืืึธืจืืืืฉ ืืืคึฟื;
- ืืึธื ื ืื ืคึผืจืืืืจื ืฆื ืึทืคึผืืึทืืืื ืืึทืกืงืก ืืึทื ืืืึทืื, ืืึธืื ืื ืงืึธืืคึผืืืืขืจ ืืึธื ืืึธืก;
- ืื ืึทืคึผื ืืึธื ืืืื ืคึผืฉืื ืคึฟืึทืจ ืื ืื ืืื ืืืืืืงืืึทื ืคืื ื ืืฆื;
- ืืืื ืืืื ืืึทืื ืืขื ืขื ืืขืืื ื ืฆืืืขืืจืืื, ืงืึทืืฉืื ื ืืืขื ืืืื ืืจืื ื!
ืืงืืจ: www.habr.com