ื•ื•ื™ ืฆื• ืžืึทื›ืŸ ื“ื™ื™ืŸ ืื™ื™ื’ืขื ืข ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืคึฟืึทืจ ืึท ืงื ื•ื™ืœ

ื”ืขืœื! ืžื™ืจ ื‘ืึทืŸ ืžืขื ื˜ืฉืŸ ืฆื• ืึทืจื‘ืขื˜ืŸ ืžื™ื˜ ื’ืจื•ื™ืก ื“ืึทื˜ืŸ. ืขืก ืื™ื– ืื•ืžืžืขื’ืœืขืš ืฆื• ื™ืžืึทื“ื–ืฉืึทืŸ ืึท ื‘ื™ืœื“ื•ื ื’ืงืจื™ื™ื– ืคึผืจืึธื’ืจืึทื ืื•ื™ืฃ ื’ืจื•ื™ืก ื“ืึทื˜ืŸ ืึธืŸ ืึทืŸ ืื™ื™ื’ืขื ืข ืงื ื•ื™ืœ, ืื•ื™ืฃ ื•ื•ืึธืก ืึทืœืข ืคึผืึทืจื˜ื™ืกืึทืคึผืึทื ืฅ ืึทืจื‘ืขื˜ ืฆื•ื–ืึทืžืขืŸ. ืคึฟืึทืจ ื“ืขื ืกื™ื‘ื”, ืื•ื ื“ื–ืขืจ ืคึผืจืึธื’ืจืึทื ืฉื˜ืขื ื“ื™ืง ื”ืื˜ ืขืก ๐Ÿ™‚ ืžื™ืจ ื–ืขื ืขืŸ ืคืึทืจืงื ืึทืกื˜ ืื™ืŸ ื–ื™ื™ึทืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ, ื˜ื•ื ื™ื ื’ ืื•ืŸ ืึทื“ืžื™ื ื™ืกื˜ืจืึทืฆื™ืข, ืื•ืŸ ื“ื™ ื’ื™ื™ื– ื’ืœื™ื™ืš ืงืึทื˜ืขืจ MapReduce ื“ื–ืฉืึธื‘ืก ื“ืึธืจื˜ ืื•ืŸ ื ื•ืฆืŸ ืกืคึผืึทืจืง.

ืื™ืŸ ื“ืขื ืคึผืึธืกื˜ืŸ ืžื™ืจ ื•ื•ืขื˜ ื–ืึธื’ืŸ ืื™ืจ ื•ื•ื™ ืžื™ืจ ืกืึทืœื•ื•ื“ ื“ื™ ืคึผืจืึธื‘ืœืขื ืคื•ืŸ ืึทื ื™ื•ื•ืึทืŸ ืงื ื•ื™ืœ ืœืึธื•ื“ื™ื ื’ ื“ื•ืจืš ืฉืจื™ื™ื‘ืŸ ืื•ื ื“ื–ืขืจ ืื™ื™ื’ืขื ืข ืึทื•ื˜ืึธืกืงืึทืœืขืจ ื ื™ืฆืŸ ื“ื™ ื•ื•ืึธืœืงืŸ Mail.ru ืงืœืึธื•ื“ ืกืึทืœื•ืฉืึทื ื–.

ืคึผืจืึธื‘ืœืขื

ืื•ื ื“ื–ืขืจ ืงื ื•ื™ืœ ืื™ื– ื ื™ืฉื˜ ื’ืขื ื™ืฆื˜ ืื™ืŸ ืึท ื˜ื™ืคึผื™ืฉ ืžืึธื“ืข. ื‘ืึทื–ื™ื™ึทื˜ื™ืงื•ื ื’ ืื™ื– ื”ืขื›ืกื˜ ืึทื ื™ื•ื•ืึทืŸ. ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ืขืก ื–ืขื ืขืŸ ืคึผืจืึทืงื˜ื™ืฉ ืงืœืืกืŸ, ื•ื•ืขืŸ ืึทืœืข 30 ืžืขื ื˜ืฉืŸ ืื•ืŸ ืึท ืœืขืจืขืจ ื’ื™ื™ืŸ ืฆื• ื“ื™ ืงื ื•ื™ืœ ืื•ืŸ ืึธื ื”ื™ื™ื‘ืŸ ืฆื• ื ื•ืฆืŸ ืขืก. ืึธื“ืขืจ ื•ื•ื™ื“ืขืจ, ืขืก ื–ืขื ืขืŸ ื˜ืขื’ ืื™ื™ื“ืขืจ ื“ื™ ื˜ืขืจืžื™ืŸ ื•ื•ืขืŸ ื“ื™ ืžืึทืกืข ื™ื ืงืจื™ืกื™ื– ื–ื™ื™ืขืจ. ื“ื™ ืžื ื•ื—ื” ืคื•ืŸ ื“ื™ ืฆื™ื™ื˜, ื“ื™ ืงื ื•ื™ืœ ืึทืคึผืขืจื™ื™ืฅ ืื™ืŸ ื•ื ื“ืขืจืœืึธืึทื“ ืžืึธื“ืข.

ืœื™ื™ื–ื•ื ื’ #1 ืื™ื– ืฆื• ื”ืึทืœื˜ืŸ ืึท ืงื ื•ื™ืœ ื•ื•ืึธืก ื•ื•ืขื˜ ื•ื•ื™ื˜ืกื˜ืึทื ื“ ืฉืคึผื™ืฅ ืœืึธื•ื“ื–, ืึธื‘ืขืจ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืœื™ื™ื“ื™ืง ื“ื™ ืžื ื•ื—ื” ืคื•ืŸ ื“ื™ ืฆื™ื™ื˜.

ืœื™ื™ื–ื•ื ื’ #2 ืื™ื– ืฆื• ื”ืึทืœื˜ืŸ ืึท ืงืœื™ื™ืŸ ืงื ื•ื™ืœ, ืฆื• ื•ื•ืึธืก ืื™ืจ ืžืึทื ื™ื•ืึทืœื™ ืœื™ื™ื’ืŸ ื ืึธื•ื“ื– ืื™ื™ื“ืขืจ ืงืœืืกืŸ ืื•ืŸ ื‘ืขืฉืึทืก ืฉืคึผื™ืฅ ืœืึธื•ื“ื–.

ืœื™ื™ื–ื•ื ื’ #3 ืื™ื– ืฆื• ื”ืึทืœื˜ืŸ ืึท ืงืœื™ื™ืŸ ืงื ื•ื™ืœ ืื•ืŸ ืฉืจื™ื™ึทื‘ืŸ ืึท ืึทื•ื˜ืึธืกืงืึทืœืขืจ ื•ื•ืึธืก ื•ื•ืขื˜ ืžืึธื ื™ื˜ืึธืจ ื“ื™ ืงืจืึทื ื˜ ืžืึทืกืข ืคื•ืŸ โ€‹โ€‹ื“ืขื ืงื ื•ื™ืœ ืื•ืŸ, ื ื™ืฆืŸ ืคืึทืจืฉื™ื“ืŸ ืึทืคึผื™ืก, ืœื™ื™ื’ืŸ ืื•ืŸ ืึทืจืึธืคึผื ืขืžืขืŸ ื ืึธื•ื“ื– ืคื•ืŸ ื“ืขื ืงื ื•ื™ืœ.

ืื™ืŸ ื“ืขื ืคึผืึธืกื˜ืŸ ืžื™ืจ ื•ื•ืขืœืŸ ืจืขื“ืŸ ื•ื•ืขื’ืŸ ืœื™ื™ื–ื•ื ื’ #3. ื“ืขืจ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืื™ื– ื”ืขื›ืกื˜ ืึธืคืขื ื’ื™ืง ืื•ื™ืฃ ืคื•ื ื“ืจื•ื™ืกื ื“ื™ืง ืกื™ื‘ื•ืช ืืœื ื•ื•ื™ ื™ื ืขืจืœืขืš ืึธื ืขืก, ืื•ืŸ ืคึผืจืึทื•ื•ื™ื™ื“ืขืจื– ืึธืคื˜ ื˜ืึธืŸ ื ื™ื˜ ืฆื•ืฉื˜ืขืœืŸ ืขืก. ืžื™ืจ ื ื•ืฆืŸ ื“ื™ Mail.ru ืงืœืึธื•ื“ ืกืึทืœื•ืฉืึทื ื– ื•ื•ืึธืœืงืŸ ื™ื ืคืจืึทืกื˜ืจืึทืงื˜ืฉืขืจ ืื•ืŸ ื’ืขืฉืจื™ื‘ืŸ ืึทืŸ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืžื™ื˜ ื“ื™ MCS API. ืื•ืŸ ื–ื™ื ื˜ ืžื™ืจ ืœืขืจื ืขืŸ ื•ื•ื™ ืฆื• ืึทืจื‘ืขื˜ืŸ ืžื™ื˜ ื“ืึทื˜ืŸ, ืžื™ืจ ื‘ืึทืฉืœืึธืกืŸ ืฆื• ื•ื•ื™ื™ึทื–ืŸ ื•ื•ื™ ืื™ืจ ืงืขื ืขืŸ ืฉืจื™ื™ึทื‘ืŸ ืึท ืขื ืœืขืš ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืคึฟืึทืจ ื“ื™ื™ืŸ ืื™ื™ื’ืขื ืข ืฆื•ื•ืขืงืŸ ืื•ืŸ ื ื•ืฆืŸ ืขืก ืžื™ื˜ ื“ื™ื™ืŸ ื•ื•ืึธืœืงืŸ.

ืคึผืจืขืจืขืงื•ื•ื™ืกื™ื˜ืขืก

ืขืจืฉื˜ืขืจ, ืื™ืจ ืžื•ื–ืŸ ื”ืึธื‘ืŸ ืึท Hadoop ืงื ื•ื™ืœ. ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ืžื™ืจ ื ื•ืฆืŸ ื“ื™ HDP ืคืึทืจืฉืคึผืจื™ื™ื˜ื•ื ื’.

ื›ึผื“ื™ ื“ื™ื™ืŸ ื ืึธื•ื“ื– ื’ืขืฉื•ื•ื™ื ื“ ืฆื•ื’ืขืœื™ื™ื’ื˜ ืื•ืŸ ืึทื•ื•ืขืงื’ืขื ื•ืžืขืŸ, ืื™ืจ ืžื•ื–ืŸ ื”ืึธื‘ืŸ ืึท ื–ื™ื›ืขืจ ืคืึทืจืฉืคึผืจื™ื™ื˜ื•ื ื’ ืคื•ืŸ ืจืึธืœืขืก ืฆื•ื•ื™ืฉืŸ ื“ื™ ื ืึธื•ื“ื–.

  1. ื‘ืขืœ ื ืึธื“ืข. ื ื•, ืขืก ืื™ื– ื ื™ื˜ ื“ืึทืจืคึฟืŸ ืฆื• ื“ืขืจืงืœืขืจืŸ ืขืคึผืขืก ืกืคึผืขืฆื™ืขืœ: ื“ื™ ื”ื•ื™ืคึผื˜ ื ืึธื“ืข ืคื•ืŸ โ€‹โ€‹ื“ืขื ืงื ื•ื™ืœ, ืื•ื™ืฃ ื•ื•ืึธืก, ืœืžืฉืœ, ื“ืขืจ ืึธื ืฆื™ื ื“ืŸ ืฉืึธืคืขืจ ืื™ื– ืœืึธื ื˜ืฉื˜, ืื•ื™ื‘ ืื™ืจ ื ื•ืฆืŸ ื“ื™ ื™ื ื˜ืขืจืึทืงื˜ื™ื•ื• ืžืึธื“ืข.
  2. ื“ืึทื˜ืข ื ืึธื“ืข. ื“ืึธืก ืื™ื– ื“ืขืจ ื ืึธื“ืข ืื•ื™ืฃ ื•ื•ืึธืก ืื™ืจ ืงืจืึธื ื“ืึทื˜ืŸ ืื•ื™ืฃ HDFS ืื•ืŸ ื•ื•ื• ื—ืฉื‘ื•ื ื•ืช ื ืขืžืขืŸ ืึธืจื˜.
  3. ืงืึทืžืคึผื™ื•ื˜ื™ื ื’ ื ืึธื“ืข. ื“ืึธืก ืื™ื– ืึท ื ืึธื“ืข ื•ื•ื• ืื™ืจ ื˜ืึธืŸ ื ื™ื˜ ืงืจืึธื ืขืคึผืขืก ืื•ื™ืฃ HDFS, ืึธื‘ืขืจ ื•ื•ื• ื—ืฉื‘ื•ื ื•ืช ืคึผืึทืกื™ืจืŸ.

ื•ื•ื™ื›ื˜ื™ืง ืคื•ื ื˜. ืึทื•ื˜ืึธืกืงืึทืœื™ื ื’ ื•ื•ืขื˜ ืคึผืึทืกื™ืจืŸ ืจืขื›ื˜ ืฆื• ื ืึธื•ื“ื– ืคื•ืŸ ื“ื™ ื“ืจื™ื˜ ื˜ื™ืคึผ. ืื•ื™ื‘ ืื™ืจ ืึธื ื”ื™ื™ื‘ืŸ ืฆื• ื ืขืžืขืŸ ืื•ืŸ ืœื™ื™ื’ืŸ ื ืึธื•ื“ื– ืคื•ืŸ ื“ื™ ืจื’ืข ื˜ื™ืคึผ, ื“ื™ ืขื ื˜ืคืขืจ ื’ื™ื›ืงื™ื™ึทื˜ ื•ื•ืขื˜ ื–ื™ื™ืŸ ื–ื™ื™ืขืจ ื ื™ื“ืขืจื™ืง - ื“ื™ืงืึทืžื™ืฉืึทื ื™ื ื’ ืื•ืŸ ืจื™ืงืึธืžืžื™ื˜ื™ื ื’ ื•ื•ืขื˜ ื ืขืžืขืŸ ืฉืขื” ืื•ื™ืฃ ื“ื™ื™ืŸ ืงื ื•ื™ืœ. ื“ืึธืก, ืคื•ืŸ ืงื•ืจืก, ืื™ื– ื ื™ืฉื˜ ื•ื•ืึธืก ืื™ืจ ื“ืขืจื•ื•ืึทืจื˜ืŸ ืคื•ืŸ ืึทื•ื˜ืึธืกืงืึทืœื™ื ื’. ืึทื– ืื™ื–, ืžื™ืจ ื˜ืึธืŸ ื ื™ื˜ ืึธื ืจื™ืจืŸ ื ืึธื•ื“ื– ืคื•ืŸ ื“ืขืจ ืขืจืฉื˜ืขืจ ืื•ืŸ ืจื’ืข ื˜ื™ืคึผ. ื–ื™ื™ ื•ื•ืขืœืŸ ืคืึธืจืฉื˜ืขืœืŸ ืึท ืžื™ื ื™ืžื•ื ื•ื•ื™ื™ืึทื‘ืึทืœ ืงื ื•ื™ืœ ื•ื•ืึธืก ื•ื•ืขื˜ ืขืงืกื™ืกื˜ื™ืจืŸ ืื™ื‘ืขืจ ื“ืขืจ ื’ืขื“ื•ื™ืขืจ ืคื•ืŸ ื“ื™ ืคึผืจืึธื’ืจืึทื.

ืึทื–ื•ื™, ืื•ื ื“ื–ืขืจ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืื™ื– ื’ืขืฉืจื™ื‘ืŸ ืื™ืŸ Python 3, ื ื™ืฆื˜ ื“ื™ Ambari API ืฆื• ืคื™ืจืŸ ืงื ื•ื™ืœ ื‘ืึทื“ื™ื ื•ื ื’ืก, ื ื™ืฆื˜ ืึทืคึผื™ ืคึฟื•ืŸ Mail.ru ืงืœืึธื•ื“ ืกืึทืœื•ืฉืึทื ื– (MCS) ืคึฟืึทืจ ืกื˜ืึทืจื˜ื™ื ื’ ืื•ืŸ ืกื˜ืึธืคึผืคึผื™ื ื’ ืžืืฉื™ื ืขืŸ.

ืœื™ื™ื–ื•ื ื’ ืึทืจืงืึทื˜ืขืงื˜ืฉืขืจ

  1. ืžืึธื“ื•ืœืข autoscaler.py. ืขืก ื›ึผื•ืœืœ ื“ืจื™ื™ ืงืœืืกืŸ: 1) ืคืึทื ื’ืงืฉืึทื ื– ืคึฟืึทืจ ืืจื‘ืขื˜ืŸ ืžื™ื˜ Ambari, 2) ืคืึทื ื’ืงืฉืึทื ื– ืคึฟืึทืจ ืืจื‘ืขื˜ืŸ ืžื™ื˜ MCS, 3) ืคืึทื ื’ืงืฉืึทื ื– ื’ืœื™ื™ืš ืฉื™ื™ึทื›ื•ืช ืฆื• ื“ื™ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืœืึธื’ื™ืง.
  2. ืฉืจื™ืคื˜ observer.py. ื™ืกืขื ืฉืึทืœื™ ืขืก ื‘ืืฉื˜ื™ื™ื˜ ืคื•ืŸ ืคืึทืจืฉื™ื“ืขื ืข ื›ึผืœืœื™ื: ื•ื•ืขืŸ ืื•ืŸ ืื™ืŸ ื•ื•ืึธืก ืžืึธื•ืžืึทื ืฅ ืฆื• ืจื•ืคืŸ ื“ื™ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืคืึทื ื’ืงืฉืึทื ื–.
  3. ืงืึธื ืคื™ื’ื•ืจืึทื˜ื™ืึธืŸ ื˜ืขืงืข config.py. ืขืก ื›ึผื•ืœืœ, ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ืึท ืจืฉื™ืžื” ืคื•ืŸ ื ืึธื•ื“ื– ืขืจืœื•ื™ื‘ื˜ ืคึฟืึทืจ ืึทื•ื˜ืึธืกืงืึทืœื™ื ื’ ืื•ืŸ ืื ื“ืขืจืข ืคึผืึทืจืึทืžืขื˜ืขืจืก ื•ื•ืึธืก ื•ื•ื™ืจืงืŸ, ืœืžืฉืœ, ื•ื•ื™ ืœืึทื ื’ ืฆื• ื•ื•ืึทืจื˜ืŸ ืคึฟื•ืŸ ื“ืขื ืžืึธืžืขื ื˜ ืึท ื ื™ื™ึท ื ืึธื“ืข ืื™ื– ืฆื•ื’ืขื’ืขื‘ืŸ. ืขืก ื–ืขื ืขืŸ ืื•ื™ืš ื˜ื™ืžืขืกื˜ืึทืžืคึผืก ืคึฟืึทืจ ื“ื™ ืึธื ื”ื™ื™ื‘ ืคื•ืŸ ืงืœืืกืŸ, ืึทื–ื•ื™ ืึทื– ืื™ื™ื“ืขืจ ื“ื™ ืงืœืึทืก ื“ื™ ืžืึทืงืกื™ืžื•ื ื“ืขืจืœื•ื™ื‘ื˜ ืงื ื•ื™ืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืื™ื– ืœืึธื ื˜ืฉื˜.

ืœืึธืžื™ืจ ืื™ืฆื˜ ืงื•ืงืŸ ืื™ืŸ ื“ื™ ืฉื˜ื™ืง ืคื•ืŸ ืงืึธื“ ืื™ืŸ ื“ื™ ืขืจืฉื˜ืขืจ ืฆื•ื•ื™ื™ ื˜ืขืงืขืก.

1. Autoscaler.py ืžืึธื“ื•ืœืข

ืึทืžื‘ืึทืจื™ ืงืœืึทืก

ื“ืึธืก ืื™ื– ื•ื•ื™ ืึท ืฉื˜ื™ืง ืคื•ืŸ ืงืึธื“ ืžื™ื˜ ืึท ืงืœืึทืก ืงื•ืงื˜ ื•ื•ื™ Ambari:

class Ambari:
    def __init__(self, ambari_url, cluster_name, headers, auth):
        self.ambari_url = ambari_url
        self.cluster_name = cluster_name
        self.headers = headers
        self.auth = auth

    def stop_all_services(self, hostname):
        url = self.ambari_url + self.cluster_name + '/hosts/' + hostname + '/host_components/'
        url2 = self.ambari_url + self.cluster_name + '/hosts/' + hostname
        req0 = requests.get(url2, headers=self.headers, auth=self.auth)
        services = req0.json()['host_components']
        services_list = list(map(lambda x: x['HostRoles']['component_name'], services))
        data = {
            "RequestInfo": {
                "context":"Stop All Host Components",
                "operation_level": {
                    "level":"HOST",
                    "cluster_name": self.cluster_name,
                    "host_names": hostname
                },
                "query":"HostRoles/component_name.in({0})".format(",".join(services_list))
            },
            "Body": {
                "HostRoles": {
                    "state":"INSTALLED"
                }
            }
        }
        req = requests.put(url, data=json.dumps(data), headers=self.headers, auth=self.auth)
        if req.status_code in [200, 201, 202]:
            message = 'Request accepted'
        else:
            message = req.status_code
        return message

ืื•ื™ื‘ืŸ, ื•ื•ื™ ืึท ื‘ื™ื™ึทืฉืคึผื™ืœ, ืื™ืจ ืงืขื ืขืŸ ื–ืขืŸ ื“ื™ ื™ืžืคึผืœืึทืžืขื ื˜ื™ื™ืฉืึทืŸ ืคื•ืŸ ื“ื™ ืคื•ื ืงืฆื™ืข stop_all_services, ื•ื•ืึธืก ืกื˜ืึทืคึผืก ืึทืœืข ืกืขืจื•ื•ื™ืกืขืก ืื•ื™ืฃ ื“ื™ ื’ืขื‘ืขื˜ืŸ ืงื ื•ื™ืœ ื ืึธื“ืข.

ื‘ืฒ ื ืืจืฒื ื’ืื  ื’ ืฆื• ื ืงืœืืก Ambari ื“ื• ืคืืจื‘ื™ื™:

  • ambari_url, ืœืžืฉืœ, ื•ื•ื™ 'http://localhost:8080/api/v1/clusters/',
  • cluster_name - ื“ืขืจ ื ืึธืžืขืŸ ืคื•ืŸ ื“ื™ื™ืŸ ืงื ื•ื™ืœ ืื™ืŸ Ambari,
  • headers = {'X-Requested-By': 'ambari'}
  • ืื•ืŸ ืื™ื ืขื•ื•ื™ื™ื ื™ืง auth ื“ืึธ ืื™ื– ื“ื™ื™ืŸ ื ืืžืขืŸ ืื•ืŸ ืคึผืึทืจืึธืœ ืคึฟืึทืจ Ambari: auth = ('login', 'password').

ื“ื™ ืคื•ื ืงืฆื™ืข ื–ื™ืš ืื™ื– ื’ืึธืจื ื™ืฉื˜ ืžืขืจ ื•ื•ื™ ืึท ืคึผืึธืจ ืคื•ืŸ ืงืึทืœืœืก ื“ื•ืจืš ื“ื™ REST API ืฆื• Ambari. ืคึฟื•ืŸ ืึท ืœืึทื“ื–ืฉื™ืงืึทืœ ืคื•ื ื˜ ืคื•ืŸ ืžื™ื™ื ื•ื ื’, ืžื™ืจ ืขืจืฉื˜ืขืจ ื‘ืึทืงื•ืžืขืŸ ืึท ืจืฉื™ืžื” ืคื•ืŸ ืคืœื™ืกื ื“ื™ืง ื‘ืึทื“ื™ื ื•ื ื’ืก ืื•ื™ืฃ ืึท ื ืึธื“ืข, ืื•ืŸ ื“ืึทืŸ ื‘ืขื˜ืŸ ืื•ื™ืฃ ืึท ื’ืขื’ืขื‘ืŸ ืงื ื•ื™ืœ, ืื•ื™ืฃ ืึท ื’ืขื’ืขื‘ืŸ ื ืึธื“ืข, ืฆื• ืึทืจื™ื‘ืขืจืคื™ืจืŸ ื‘ืึทื“ื™ื ื•ื ื’ืขืŸ ืคื•ืŸ ื“ืขืจ ืจืฉื™ืžื” ืฆื• ื“ื™ ืฉื˜ืึทื˜ INSTALLED. ืคืึทื ื’ืงืฉืึทื ื– ืคึฟืึทืจ ืœืึธื ื˜ืฉื™ื ื’ ืึทืœืข ื‘ืึทื“ื™ื ื•ื ื’ืก, ืคึฟืึทืจ ื˜ืจืึทื ืกืคืขืจื™ื ื’ ื ืึธื•ื“ื– ืฆื• ืฉื˜ืึทื˜ Maintenance ืขื˜ืง ืงื•ืง ืขื ืœืขืš - ื–ื™ื™ ื–ืขื ืขืŸ ื ืึธืจ ืึท ื‘ื™ืกืœ ืจื™ืงื•ื•ืขืก ื“ื•ืจืš ื“ื™ ืึทืคึผื™.

ืงืœืึทืก ืžืงืก

ื“ืึธืก ืื™ื– ื•ื•ื™ ืึท ืฉื˜ื™ืง ืคื•ืŸ ืงืึธื“ ืžื™ื˜ ืึท ืงืœืึทืก ืงื•ืงื˜ ื•ื•ื™ Mcs:

class Mcs:
    def __init__(self, id1, id2, password):
        self.id1 = id1
        self.id2 = id2
        self.password = password
        self.mcs_host = 'https://infra.mail.ru:8774/v2.1'

    def vm_turn_on(self, hostname):
        self.token = self.get_mcs_token()
        host = self.hostname_to_vmname(hostname)
        vm_id = self.get_vm_id(host)
        mcs_url1 = self.mcs_host + '/servers/' + self.vm_id + '/action'
        headers = {
            'X-Auth-Token': '{0}'.format(self.token),
            'Content-Type': 'application/json'
        }
        data = {'os-start' : 'null'}
        mcs = requests.post(mcs_url1, data=json.dumps(data), headers=headers)
        return mcs.status_code

ื‘ืฒ ื ืืจืฒื ื’ืื  ื’ ืฆื• ื ืงืœืืก Mcs ืžื™ืจ ืคืึธืจืŸ ื“ื™ ืคึผืจื•ื™ืขืงื˜ ืฉื™ื™ึทืŸ ืื™ืŸ ื“ื™ ื•ื•ืึธืœืงืŸ ืื•ืŸ ื“ื™ ื‘ืึทื ื™ืฆืขืจ ืฉื™ื™ึทืŸ, ื•ื•ื™ ื’ืขื–ื•ื ื˜ ื•ื•ื™ ื–ื™ื™ืŸ ืคึผืึทืจืึธืœ. ืื™ืŸ ืคึฟื•ื ืงืฆื™ืข vm_turn_on ืžื™ืจ ื•ื•ื™ืœืŸ ืฆื• ื•ื•ืขื ื“ืŸ ืื•ื™ืฃ ืื™ื™ื ืขืจ ืคื•ืŸ ื“ื™ ืžืืฉื™ื ืขืŸ. ื“ื™ ืœืึธื’ื™ืง ื“ืึธ ืื™ื– ืึท ื‘ื™ืกืœ ืžืขืจ ืงืึธืžืคึผืœื™ืฆื™ืจื˜. ืื™ืŸ ื“ื™ ืึธื ื”ื™ื™ื‘ ืคื•ืŸ ื“ื™ ืงืึธื“, ื“ืจื™ื™ ืื ื“ืขืจืข ืคืึทื ื’ืงืฉืึทื ื– ื–ืขื ืขืŸ ื’ืขืจื•ืคืŸ: 1) ืžื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ื‘ืึทืงื•ืžืขืŸ ืึท ืกื™ืžืขืŸ, 2) ืžื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ื’ืขืจ ื“ื™ ื”ืึธืกื˜ื ืึทืžืข ืื™ืŸ ื“ื™ ื ืึธืžืขืŸ ืคื•ืŸ ื“ื™ ืžืึทืฉื™ืŸ ืื™ืŸ MCS, 3) ื‘ืึทืงื•ืžืขืŸ ื“ื™ ืฉื™ื™ึทืŸ ืคื•ืŸ ื“ืขื ืžืึทืฉื™ืŸ. ื“ืขืจื ืึธืš, ืžื™ืจ ื ืึธืจ ืžืึทื›ืŸ ืึท ืคึผืึธืกื˜ืŸ ื‘ืขื˜ืŸ ืื•ืŸ ืงืึทื˜ืขืจ ื“ืขื ืžืึทืฉื™ืŸ.

ื“ืึธืก ืื™ื– ื•ื•ื™ ื“ื™ ืคื•ื ืงืฆื™ืข ืฆื• ื‘ืึทืงื•ืžืขืŸ ืึท ืกื™ืžืขืŸ ืงื•ืงื˜ ื•ื•ื™:

def get_mcs_token(self):
        url = 'https://infra.mail.ru:35357/v3/auth/tokens?nocatalog'
        headers = {'Content-Type': 'application/json'}
        data = {
            'auth': {
                'identity': {
                    'methods': ['password'],
                    'password': {
                        'user': {
                            'id': self.id1,
                            'password': self.password
                        }
                    }
                },
                'scope': {
                    'project': {
                        'id': self.id2
                    }
                }
            }
        }
        params = (('nocatalog', ''),)
        req = requests.post(url, data=json.dumps(data), headers=headers, params=params)
        self.token = req.headers['X-Subject-Token']
        return self.token

ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืงืœืึทืก

ื“ืขืจ ืงืœืึทืก ื›ึผื•ืœืœ ืคืึทื ื’ืงืฉืึทื ื– ืฉื™ื™ึทื›ื•ืช ืฆื• ื“ื™ ืึธืคึผืขืจื™ื™ื˜ื™ื ื’ ืœืึธื’ื™ืง ื–ื™ืš.

ื“ืึธืก ืื™ื– ื•ื•ื™ ืึท ืฉื˜ื™ืง ืคื•ืŸ ืงืึธื“ ืคึฟืึทืจ ื“ืขื ืงืœืึทืก ืงื•ืงื˜ ื•ื•ื™:

class Autoscaler:
    def __init__(self, ambari, mcs, scaling_hosts, yarn_ram_per_node, yarn_cpu_per_node):
        self.scaling_hosts = scaling_hosts
        self.ambari = ambari
        self.mcs = mcs
        self.q_ram = deque()
        self.q_cpu = deque()
        self.num = 0
        self.yarn_ram_per_node = yarn_ram_per_node
        self.yarn_cpu_per_node = yarn_cpu_per_node

    def scale_down(self, hostname):
        flag1 = flag2 = flag3 = flag4 = flag5 = False
        if hostname in self.scaling_hosts:
            while True:
                time.sleep(5)
                status1 = self.ambari.decommission_nodemanager(hostname)
                if status1 == 'Request accepted' or status1 == 500:
                    flag1 = True
                    logging.info('Decomission request accepted: {0}'.format(flag1))
                    break
            while True:
                time.sleep(5)
                status3 = self.ambari.check_service(hostname, 'NODEMANAGER')
                if status3 == 'INSTALLED':
                    flag3 = True
                    logging.info('Nodemaneger decommissioned: {0}'.format(flag3))
                    break
            while True:
                time.sleep(5)
                status2 = self.ambari.maintenance_on(hostname)
                if status2 == 'Request accepted' or status2 == 500:
                    flag2 = True
                    logging.info('Maintenance request accepted: {0}'.format(flag2))
                    break
            while True:
                time.sleep(5)
                status4 = self.ambari.check_maintenance(hostname, 'NODEMANAGER')
                if status4 == 'ON' or status4 == 'IMPLIED_FROM_HOST':
                    flag4 = True
                    self.ambari.stop_all_services(hostname)
                    logging.info('Maintenance is on: {0}'.format(flag4))
                    logging.info('Stopping services')
                    break
            time.sleep(90)
            status5 = self.mcs.vm_turn_off(hostname)
            while True:
                time.sleep(5)
                status5 = self.mcs.get_vm_info(hostname)['server']['status']
                if status5 == 'SHUTOFF':
                    flag5 = True
                    logging.info('VM is turned off: {0}'.format(flag5))
                    break
            if flag1 and flag2 and flag3 and flag4 and flag5:
                message = 'Success'
                logging.info('Scale-down finished')
                logging.info('Cooldown period has started. Wait for several minutes')
        return message

ืžื™ืจ ืึธื ื ืขืžืขืŸ ืงืœืืกืŸ ืคึฟืึทืจ ืคึผืึธื–ื™ืฆื™ืข. Ambari ะธ Mcs, ืึท ืจืฉื™ืžื” ืคื•ืŸ ื ืึธื•ื“ื– ื•ื•ืึธืก ื–ืขื ืขืŸ ืขืจืœื•ื™ื‘ื˜ ืคึฟืึทืจ ืกืงื™ื™ืœื™ื ื’, ื•ื•ื™ ื’ืขื–ื•ื ื˜ ื•ื•ื™ ื ืึธื“ืข ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืคึผืึทืจืึทืžืขื˜ืขืจืก: ื–ื›ึผืจื•ืŸ ืื•ืŸ ืงืคึผื• ืึทืœืึทืงื™ื™ื˜ื™ื“ ืฆื• ื“ื™ ื ืึธื“ืข ืื™ืŸ YARN. ืขืก ื–ืขื ืขืŸ ืื•ื™ืš 2 ื™ื ืขืจืœืขืš ืคึผืึทืจืึทืžืขื˜ืขืจืก q_ram, q_cpu, ื•ื•ืึธืก ื–ืขื ืขืŸ ืงื™ื•ื–. ื ื™ืฆืŸ ื–ื™ื™, ืžื™ืจ ืงืจืึธื ื“ื™ ื•ื•ืึทืœื•ืขืก ืคื•ืŸ ื“ื™ ืงืจืึทื ื˜ ืงื ื•ื™ืœ ืžืึทืกืข. ืื•ื™ื‘ ืžื™ืจ ื–ืขืŸ ืึทื– ืื™ืŸ ื“ื™ ืœืขืฆื˜ืข 5 ืžื™ื ื•ื˜ ืขืก ืื™ื– ื’ืขื•ื•ืขืŸ ืึท ืงืึทื ืกื™ืกื˜ืึทื ื˜ืœื™ ื’ืขื•ื•ืืงืกืŸ ืžืึทืกืข, ืžื™ืจ ื‘ืึทืฉืœื™ืกืŸ ืึทื– ืžื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืœื™ื™ื’ืŸ +1 ื ืึธื“ืข ืฆื• ื“ืขื ืงื ื•ื™ืœ. ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืื™ื– ืืžืช ืคึฟืึทืจ ื“ื™ ืงื ื•ื™ืœ ื•ื ื“ืขืจื™ื•ื˜ื™ืœื™ื–ืึทื˜ื™ืึธืŸ ืฉื˜ืึทื˜.

ื“ืขืจ ืงืึธื“ ืื•ื™ื‘ืŸ ืื™ื– ืึท ื‘ื™ื™ืฉืคึผื™ืœ ืคื•ืŸ ืึท ืคึฟื•ื ืงืฆื™ืข ื•ื•ืึธืก ืจื™ืžื•ื•ื•ื– ืึท ืžืึทืฉื™ืŸ ืคื•ืŸ ื“ืขื ืงื ื•ื™ืœ ืื•ืŸ ืกื˜ืึทืคึผืก ืขืก ืื™ืŸ ื“ื™ ื•ื•ืึธืœืงืŸ. ืขืจืฉื˜ืขืจ ืขืก ืื™ื– ืึท ื“ื™ืงืึทืžื™ืฉืึทื ื™ื ื’ YARN Nodemanager, ื“ืขืžืึธืœื˜ ื“ืขืจ ืžืึธื“ืข ื˜ื•ืจื ืก ืื•ื™ืฃ Maintenance, ื“ืขืžืึธืœื˜ ืžื™ืจ ื”ืึทืœื˜ืŸ ืึทืœืข ืกืขืจื•ื•ื™ืกืขืก ืื•ื™ืฃ ื“ื™ ืžืึทืฉื™ืŸ ืื•ืŸ ืงืขืจ ืึทื•ื•ืขืง ื“ื™ ื•ื•ื™ืจื˜ื•ืึทืœ ืžืึทืฉื™ืŸ ืื™ืŸ ื“ื™ ื•ื•ืึธืœืงืŸ.

2. ืกืงืจื™ืคึผื˜ observer.py

ืžื•ืกื˜ืขืจ ืงืึธื“ ืคึฟื•ืŸ ื“ืึธืจื˜:

if scaler.assert_up(config.scale_up_thresholds) == True:
        hostname = cloud.get_vm_to_up(config.scaling_hosts)
        if hostname != None:
            status1 = scaler.scale_up(hostname)
            if status1 == 'Success':
                text = {"text": "{0} has been successfully scaled-up".format(hostname)}
                post = {"text": "{0}".format(text)}
                json_data = json.dumps(post)
                req = requests.post(webhook, data=json_data.encode('ascii'), headers={'Content-Type': 'application/json'})
                time.sleep(config.cooldown_period*60)

ืื™ืŸ ืขืก, ืžื™ืจ ืงืึธื ื˜ืจืึธืœื™ืจืŸ ืฆื™ ื˜ื ืึธื™ื ื–ืขื ืขืŸ ื‘ืืฉืืคืŸ ืคึฟืึทืจ ื™ื ืงืจื™ืกื™ื ื’ ื“ื™ ืงืึทืคึผืึทืฆื™ื˜ืขื˜ ืคื•ืŸ ื“ืขื ืงื ื•ื™ืœ ืื•ืŸ ืฆื™ ืขืก ื–ืขื ืขืŸ ืงื™ื™ืŸ ืžืืฉื™ื ืขืŸ ืื™ืŸ ืจืขื–ืขืจื•ื•, ื‘ืึทืงื•ืžืขืŸ ื“ื™ ื‘ืึทืœืขื‘ืึธืก ื ืึธืžืขืŸ ืคื•ืŸ ืื™ื™ื ืขืจ ืคื•ืŸ ื–ื™ื™, ืœื™ื™ื’ืŸ ืขืก ืฆื• ื“ื™ ืงื ื•ื™ืœ ืื•ืŸ ืึทืจื•ื™ืกื’ืขื‘ืŸ ืึท ืึธื ื–ืึธื’ ื•ื•ืขื’ืŸ ื“ืขื ืื™ืŸ ืื•ื ื“ื–ืขืจ ืžืึทื ืฉืึทืคึฟื˜ ืก ืกืœืึทืงืง. ื ืึธืš ื•ื•ืึธืก ืขืก ืกื˜ืึทืจืฅ cooldown_period, ื•ื•ืขืŸ ืžื™ืจ ื˜ืึธืŸ ื ื™ื˜ ืœื™ื™ื’ืŸ ืึธื“ืขืจ ืึทืจืึธืคึผื ืขืžืขืŸ ืขืคึผืขืก ืคื•ืŸ ื“ืขื ืงื ื•ื™ืœ, ืึธื‘ืขืจ ื ืึธืจ ืžืึธื ื™ื˜ืึธืจ ื“ื™ ืžืึทืกืข. ืื•ื™ื‘ ืขืก ืื™ื– ืกื˜ื™ื™ื‘ืึทืœื™ื™ื–ื“ ืื•ืŸ ืื™ื– ืื™ืŸ ื“ื™ ืงืึธืจื™ื“ืึธืจ ืคื•ืŸ ืึธืคึผื˜ื™ืžืึทืœ ืžืึทืกืข ื•ื•ืึทืœื•ืขืก, ืžื™ืจ ืคืฉื•ื˜ ืคืึธืจื–ืขืฆืŸ ืžืึธื ื™ื˜ืึธืจื™ื ื’. ืื•ื™ื‘ ืื™ื™ืŸ ื ืึธื“ืข ืื™ื– ื ื™ืฉื˜ ื’ืขื ื•ื’, ืžื™ืจ ืœื™ื™ื’ืŸ ืื ื“ืขืจืŸ.

ืคึฟืึทืจ ืงืึทืกืขืก ื•ื•ืขืŸ ืžื™ืจ ื”ืึธื‘ืŸ ืึท ืœืขืงืฆื™ืข ืคืึธืจื•ื™ืก, ืžื™ืจ ืฉื•ื™ืŸ ื•ื•ื™ืกืŸ ืคึฟืึทืจ ื–ื™ื›ืขืจ ืึทื– ืื™ื™ืŸ ื ืึธื“ืข ื•ื•ืขื˜ ื ื™ืฉื˜ ื–ื™ื™ืŸ ื’ืขื ื•ื’, ืึทื–ื•ื™ ืžื™ืจ ืžื™ื“ ืึธื ื”ื™ื™ื‘ืŸ ืึทืœืข ื“ื™ ืคึฟืจื™ื™ึท ื ืึธื•ื“ื– ืื•ืŸ ื”ืึทืœื˜ืŸ ื–ื™ื™ ืึทืงื˜ื™ื•ื• ื‘ื™ื– ื“ืขื ืกื•ืฃ ืคื•ืŸ ื“ื™ ืœืขืงืฆื™ืข. ื“ืึธืก ื›ืึทืคึผืึทื ื– ื ื™ืฆืŸ ืึท ืจืฉื™ืžื” ืคื•ืŸ ืึทืงื˜ื™ื•ื•ื™ื˜ืขื˜ ื˜ื™ืžืขืกื˜ืึทืžืคึผืก.

ืกืึธืฃ

ืึทื•ื˜ืึธืกืงืึทืœืขืจ ืื™ื– ืึท ื’ื•ื˜ ืื•ืŸ ื‘ืึทืงื•ื•ืขื ืœื™ื™ื–ื•ื ื’ ืคึฟืึทืจ ื“ื™ ืงืึทืกืขืก ื•ื•ืขืŸ ืื™ืจ ื“ืขืจืคืึทืจื•ื ื’ ืึทื ื™ื•ื•ืึทืŸ ืงื ื•ื™ืœ ืœืึธื•ื“ื™ื ื’. ืื™ืจ ืกื™ื™ืžืึทืœื˜ื™ื™ื ื™ืึทืกืœื™ ื“ืขืจื’ืจื™ื™ื›ืŸ ื“ื™ ื’ืขื‘ืขื˜ืŸ ืงื ื•ื™ืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืคึฟืึทืจ ืฉืคึผื™ืฅ ืœืึธื•ื“ื– ืื•ืŸ ืื™ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืฆื™ื™ื˜ ื˜ืึธืŸ ื ื™ื˜ ื”ืึทืœื˜ืŸ ื“ืขื ืงื ื•ื™ืœ ื‘ืขืฉืึทืก ื•ื ื“ืขืจืœืึธืึทื“, ืฉืคึผืึธืจืŸ ื’ืขืœื˜. ื ื•, ืคึผืœื•ืก ื“ืึธืก ืึทืœืฅ ื›ืึทืคึผืึทื ื– ืื•ื™ื˜ืึธืžืึทื˜ื™ืฉ ืึธืŸ ื“ื™ื™ืŸ ืึธื ื˜ื™ื™ืœ. ื“ื™ ืึทื•ื˜ืึธืกืงืึทืœืขืจ ื–ื™ืš ืื™ื– ื’ืึธืจื ื™ืฉื˜ ืžืขืจ ื•ื•ื™ ืึท ืกื›ื•ื ืคื•ืŸ ืจื™ืงื•ื•ืขืก ืฆื• ื“ื™ ืงื ื•ื™ืœ ืคืึทืจื•ื•ืึทืœื˜ืขืจ ืึทืคึผื™ ืื•ืŸ ื“ื™ ื•ื•ืึธืœืงืŸ ืฉืคึผื™ื™ึทื–ืขืจ ืึทืคึผื™, ื’ืขืฉืจื™ื‘ืŸ ืœื•ื™ื˜ ืึท ื–ื™ื›ืขืจ ืœืึธื’ื™ืง. ื•ื•ืึธืก ืื™ืจ ื‘ืืฉื˜ื™ืžื˜ ื“ืึทืจืคึฟืŸ ืฆื• ื’ืขื“ืขื ืงืขืŸ ืื™ื– ื“ื™ ืึธืคึผื˜ื™ื™ืœ ืคื•ืŸ ื ืึธื•ื“ื– ืื™ืŸ 3 ื˜ื™ื™ืคึผืก, ื•ื•ื™ ืžื™ืจ ื’ืขืฉืจื™ื‘ืŸ ืคืจื™ืขืจ. ืื•ืŸ ืื™ืจ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืฆื•ืคืจื™ื“ืŸ.

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’