ProHoster > Блог > Nchịkwa > Otu esi eme autoscaler nke gị maka ụyọkọ

Otu esi eme autoscaler nke gị maka ụyọkọ

Nnọọ! Anyị na-azụ ndị mmadụ ka ha jiri nnukwu data rụọ ọrụ. Ọ gaghị ekwe omume iche n'echiche mmemme mmụta na nnukwu data na-enweghị ụyọkọ nke ya, nke ndị niile sonyere na-arụkọ ọrụ ọnụ. N'ihi nke a, mmemme anyị na-enwe ya mgbe niile :) Anyị na-etinye aka na nhazi ya, nhazi na nchịkwa ya, ụmụ okorobịa ahụ na-amalite MapReduce ọrụ n'ebe ahụ ma jiri Spark.

N'ime ọkwa a, anyị ga-agwa gị otu anyị si dozie nsogbu nke nchịkọta ụyọkọ na-enweghị isi site na iji igwe ojii dee autoscaler nke anyị. Mail.ru Cloud Solutions.

nsogbu

A naghị eji ụyọkọ anyị eme ihe n'ụdị ụdị. Ntufu bụ nke ukwuu enweghị isi. Dịka ọmụmaatụ, enwere klaasị bara uru, mgbe mmadụ 30 niile na onye nkuzi ga-aga ụyọkọ wee malite iji ya. Ma ọ bụ ọzọ, enwere ụbọchị tupu oge ngwụcha mgbe ibu na-abawanye nke ukwuu. Oge fọdụrụ ka ụyọkọ ahụ na-arụ ọrụ na ọnọdụ eburu ibu.

Ngwọta #1 bụ idowe ụyọkọ nke ga-anagide ibu dị elu, mana ọ ga-anọkwa n'ọrụ oge fọdụrụnụ.

Ngwọta #2 bụ idobe obere ụyọkọ, nke ị na-eji aka tinye ọnụ tupu klaasị na n'oge oke ibu.

Ngwọta #3 bụ idobe obere ụyọkọ ma dee autoscaler nke ga-enyocha ibu dị ugbu a nke ụyọkọ ahụ na, iji API dị iche iche, gbakwunye ma wepụ ọnụ na ụyọkọ ahụ.

Na nke a post anyị ga-ekwu maka ngwọta #3. Nke a autoscaler na-adabere na ihe ndị dị na mpụga karịa nke dị n'ime, na ndị na-enye ya anaghị enyekarị ya. Anyị na-eji akụrụngwa igwe ojii Mail.ru Cloud Solutions wee dee autoscaler site na iji MCS API. Ma ebe anyị na-akụzi otu esi arụ ọrụ na data, anyị kpebiri igosi otu ị nwere ike isi dee autoscaler yiri nke ahụ maka ebumnuche nke gị wee jiri igwe ojii gị jiri ya.

Prerequisites

Nke mbụ, ị ga-enwerịrị ụyọkọ Hadoop. Dịka ọmụmaatụ, anyị na-eji nkesa HDP.

Ka ọnụ gị wee tinye ngwa ngwa ma wepụ ya, ị ga-enwerịrị nkesa ụfọdụ nke ọrụ n'etiti ọnụ.

  1. Nna ukwu ọnụ. Ọfọn, ọ dịghị ihe ọ bụla dị mkpa ịkọwa ebe a: isi ọnụ nke ụyọkọ ahụ, nke, dịka ọmụmaatụ, a na-arụ ọrụ ụgbọ ala Spark, ma ọ bụrụ na ị na-eji ọnọdụ mmekọrịta.
  2. Ọnụ ụbọchị. Nke a bụ ọnụ ebe ị na-echekwa data na HDFS yana ebe mgbako na-ewere ọnọdụ.
  3. Ọnụ kọmpụta. Nke a bụ ọnụ ebe ị naghị echekwa ihe ọ bụla na HDFS, mana ebe mgbako na-eme.

Isi ihe dị mkpa. Autoscaling ga-eme n'ihi ọnụ nke ụdị nke atọ. Ọ bụrụ na ịmalite iwere na ịgbakwunye ọnụ nke ụdị nke abụọ, ọsọ nzaghachi ga-adị ntakịrị - nkwụsịtụ na ịmegharị ga-ewe awa na ụyọkọ gị. Nke a, n'ezie, abụghị ihe ị na-atụ anya site na autoscaling. Ya bụ, anyị anaghị emetụ ọnụ ụzọ nke mbụ na nke abụọ aka. Ha ga-anọchi anya ụyọkọ kacha nta ga-adị n'ime oge mmemme ahụ.

Yabụ, edere autoscaler anyị na Python 3, na-eji Ambari API jikwaa ọrụ ụyọkọ, ojiji API sitere na Mail.ru Cloud Solutions (MCS) maka ịmalite na nkwụsị igwe.

Ngwọta ije

  1. Daalụ autoscaler.py. Ọ nwere klas atọ: 1) ọrụ maka ịrụ ọrụ na Ambari, 2) ọrụ maka ịrụ ọrụ na MCS, 3) ọrụ metụtara kpọmkwem na mgbagha nke autoscaler.
  2. Ederede observer.py. N'ikpeazụ, ọ nwere iwu dị iche iche: mgbe na oge ole ị ga-akpọ ọrụ autoscaler.
  3. faịlụ nhazi config.py. Ọ nwere, dịka ọmụmaatụ, ndepụta nke ọnụ ụzọ kwere maka autoscaling na paramita ndị ọzọ na-emetụta, dịka ọmụmaatụ, ogologo oge ị ga-echere site na mgbe agbakwunyere ọnụ ọhụrụ. Enwekwara stampụ oge maka mmalite klaasị, nke mere na tupu klaasị amalitela nhazi ụyọkọ ikike ikike kacha.

Ka anyị leba anya na mpempe koodu dị n'ime faịlụ abụọ mbụ.

1. Autoscaler.py modul

klaasị Ambari

Nke a bụ otu mpempe koodu nwere klas dị ka Ambari:

class Ambari:
    def __init__(self, ambari_url, cluster_name, headers, auth):
        self.ambari_url = ambari_url
        self.cluster_name = cluster_name
        self.headers = headers
        self.auth = auth

    def stop_all_services(self, hostname):
        url = self.ambari_url + self.cluster_name + '/hosts/' + hostname + '/host_components/'
        url2 = self.ambari_url + self.cluster_name + '/hosts/' + hostname
        req0 = requests.get(url2, headers=self.headers, auth=self.auth)
        services = req0.json()['host_components']
        services_list = list(map(lambda x: x['HostRoles']['component_name'], services))
        data = {
            "RequestInfo": {
                "context":"Stop All Host Components",
                "operation_level": {
                    "level":"HOST",
                    "cluster_name": self.cluster_name,
                    "host_names": hostname
                },
                "query":"HostRoles/component_name.in({0})".format(",".join(services_list))
            },
            "Body": {
                "HostRoles": {
                    "state":"INSTALLED"
                }
            }
        }
        req = requests.put(url, data=json.dumps(data), headers=self.headers, auth=self.auth)
        if req.status_code in [200, 201, 202]:
            message = 'Request accepted'
        else:
            message = req.status_code
        return message

N'elu, dịka ọmụmaatụ, ị nwere ike ilele mmejuputa ọrụ ahụ stop_all_services, nke na-akwụsị ọrụ niile na ọnụ ụyọkọ chọrọ.

N'ọnụ ụzọ klaasị Ambari ị gafere:

  • ambari_url, ọmụmaatụ, dị ka 'http://localhost:8080/api/v1/clusters/',
  • cluster_name – aha ụyọkọ gị na Ambari,
  • headers = {'X-Requested-By': 'ambari'}
  • na n'ime auth Nke a bụ nbanye na paswọọdụ gị maka Ambari: auth = ('login', 'password').

Ọrụ ahụ n'onwe ya abụghị ihe karịrị oku ole na ole site na API REST gaa Ambari. Site n'echiche ezi uche dị na ya, anyị na-ebu ụzọ nweta ndepụta nke ọrụ na-agba ọsọ na ọnụ ọnụ, wee jụọ na ụyọkọ nyere, na ọnụ ọnụ enyere, ịnyefe ọrụ site na listi gaa na steeti. INSTALLED. Ọrụ maka ịmalite ọrụ niile, maka ịnyefe ọnụ na steeti Maintenance wdg. yiri nke a - ha bụ naanị arịrịọ ole na ole site na API.

Klas Mcs

Nke a bụ otu mpempe koodu nwere klas dị ka Mcs:

class Mcs:
    def __init__(self, id1, id2, password):
        self.id1 = id1
        self.id2 = id2
        self.password = password
        self.mcs_host = 'https://infra.mail.ru:8774/v2.1'

    def vm_turn_on(self, hostname):
        self.token = self.get_mcs_token()
        host = self.hostname_to_vmname(hostname)
        vm_id = self.get_vm_id(host)
        mcs_url1 = self.mcs_host + '/servers/' + self.vm_id + '/action'
        headers = {
            'X-Auth-Token': '{0}'.format(self.token),
            'Content-Type': 'application/json'
        }
        data = {'os-start' : 'null'}
        mcs = requests.post(mcs_url1, data=json.dumps(data), headers=headers)
        return mcs.status_code

N'ọnụ ụzọ klaasị Mcs anyị na-agafe id oru ngo n'ime igwe ojii na id onye ọrụ, yana paswọọdụ ya. Na ọrụ vm_turn_on anyị chọrọ ịgbanwuo otu igwe. Echiche ebe a bụ ntakịrị mgbagwoju anya. Na mmalite nke koodu ahụ, a na-akpọ ọrụ atọ ọzọ: 1) anyị kwesịrị ịnweta akara ngosi, 2) anyị kwesịrị ịgbanwe aha nnabata n'ime aha igwe na MCS, 3) nweta id nke igwe a. Na-esote, anyị na-arịọ arịrịọ ma malite igwe a.

Nke a bụ ọrụ maka inweta token dị ka:

def get_mcs_token(self):
        url = 'https://infra.mail.ru:35357/v3/auth/tokens?nocatalog'
        headers = {'Content-Type': 'application/json'}
        data = {
            'auth': {
                'identity': {
                    'methods': ['password'],
                    'password': {
                        'user': {
                            'id': self.id1,
                            'password': self.password
                        }
                    }
                },
                'scope': {
                    'project': {
                        'id': self.id2
                    }
                }
            }
        }
        params = (('nocatalog', ''),)
        req = requests.post(url, data=json.dumps(data), headers=headers, params=params)
        self.token = req.headers['X-Subject-Token']
        return self.token

Klas autoscaler

Klas a nwere ọrụ metụtara mgbagha arụ ọrụ n'onwe ya.

Nke a bụ otu mpempe koodu maka klaasị a:

class Autoscaler:
    def __init__(self, ambari, mcs, scaling_hosts, yarn_ram_per_node, yarn_cpu_per_node):
        self.scaling_hosts = scaling_hosts
        self.ambari = ambari
        self.mcs = mcs
        self.q_ram = deque()
        self.q_cpu = deque()
        self.num = 0
        self.yarn_ram_per_node = yarn_ram_per_node
        self.yarn_cpu_per_node = yarn_cpu_per_node

    def scale_down(self, hostname):
        flag1 = flag2 = flag3 = flag4 = flag5 = False
        if hostname in self.scaling_hosts:
            while True:
                time.sleep(5)
                status1 = self.ambari.decommission_nodemanager(hostname)
                if status1 == 'Request accepted' or status1 == 500:
                    flag1 = True
                    logging.info('Decomission request accepted: {0}'.format(flag1))
                    break
            while True:
                time.sleep(5)
                status3 = self.ambari.check_service(hostname, 'NODEMANAGER')
                if status3 == 'INSTALLED':
                    flag3 = True
                    logging.info('Nodemaneger decommissioned: {0}'.format(flag3))
                    break
            while True:
                time.sleep(5)
                status2 = self.ambari.maintenance_on(hostname)
                if status2 == 'Request accepted' or status2 == 500:
                    flag2 = True
                    logging.info('Maintenance request accepted: {0}'.format(flag2))
                    break
            while True:
                time.sleep(5)
                status4 = self.ambari.check_maintenance(hostname, 'NODEMANAGER')
                if status4 == 'ON' or status4 == 'IMPLIED_FROM_HOST':
                    flag4 = True
                    self.ambari.stop_all_services(hostname)
                    logging.info('Maintenance is on: {0}'.format(flag4))
                    logging.info('Stopping services')
                    break
            time.sleep(90)
            status5 = self.mcs.vm_turn_off(hostname)
            while True:
                time.sleep(5)
                status5 = self.mcs.get_vm_info(hostname)['server']['status']
                if status5 == 'SHUTOFF':
                    flag5 = True
                    logging.info('VM is turned off: {0}'.format(flag5))
                    break
            if flag1 and flag2 and flag3 and flag4 and flag5:
                message = 'Success'
                logging.info('Scale-down finished')
                logging.info('Cooldown period has started. Wait for several minutes')
        return message

Anyị na-anabata klaasị maka ntinye. Ambari и Mcs, ndepụta nke ọnụ na-ekwe ka scaling, yana node nhazi parameters: ebe nchekwa na cpu ekenyela ọnụ na YARN. Enwekwara parampat ime ime 2 q_ram, q_cpu, nke bụ ahịrị. Iji ha, anyị na-echekwa ụkpụrụ nke ibu ụyọkọ dị ugbu a. Ọ bụrụ na anyị ahụ na n'ime nkeji 5 gara aga enweela ibu na-abawanye ụba, mgbe ahụ anyị na-ekpebi na anyị kwesịrị ịgbakwunye ọnụ +1 na ụyọkọ ahụ. Otu ihe ahụ dịkwa maka steeti ejighi n'ọrụ ụyọkọ.

Koodu dị n'elu bụ ihe atụ nke ọrụ na-ewepụ igwe na ụyọkọ ahụ ma kwụsị ya n'igwe ojii. Nke mbụ, enwere mwepu YARN Nodemanager, mgbe ahụ mode na-agbanye Maintenance, mgbe ahụ, anyị kwụsịrị ọrụ niile na igwe ma gbanyụọ igwe mebere na igwe ojii.

2. Onye na-ekiri edemede.py

Koodu nlele si ebe ahụ:

if scaler.assert_up(config.scale_up_thresholds) == True:
        hostname = cloud.get_vm_to_up(config.scaling_hosts)
        if hostname != None:
            status1 = scaler.scale_up(hostname)
            if status1 == 'Success':
                text = {"text": "{0} has been successfully scaled-up".format(hostname)}
                post = {"text": "{0}".format(text)}
                json_data = json.dumps(post)
                req = requests.post(webhook, data=json_data.encode('ascii'), headers={'Content-Type': 'application/json'})
                time.sleep(config.cooldown_period*60)

N'ime ya, anyị na-enyocha ma emepụtara ọnọdụ maka ịbawanye ikike nke ụyọkọ ahụ yana ma enwere igwe ọ bụla echekwara, nweta aha nnabata nke otu n'ime ha, tinye ya na ụyọkọ ma bipụta ozi gbasara ya na Slack otu anyị. Mgbe nke ahụ gasịrị ọ na-amalite cooldown_period, mgbe anyị anaghị agbakwunye ma ọ bụ wepụ ihe ọ bụla na ụyọkọ ahụ, mana naanị nyochaa ibu ahụ. Ọ bụrụ na ọ kwụsie ike ma dị n'ime ụzọ nke ụkpụrụ ibu kachasị mma, mgbe ahụ, anyị na-aga n'ihu na-enyocha ya. Ọ bụrụ na otu ọnụ ezughị, mgbe ahụ, anyị na-agbakwunye ọzọ.

Maka ikpe mgbe anyị nwere nkuzi n'ihu, anyị amaralarị nke ọma na otu ọnụ agaghị ezuru, yabụ anyị na-amalite ozugbo oghere niile n'efu wee mee ka ha na-arụ ọrụ ruo ọgwụgwụ nke nkuzi. Nke a na-eme site na iji ndepụta oge akara ọrụ.

nkwubi

Autoscaler bụ ihe ngwọta dị mma ma dị mma maka ikpe ndị ahụ mgbe ị nwetara nchịkọta ụyọkọ na-enweghị isi. Ị na-enweta n'otu oge nhazi ụyọkọ achọrọ maka ibu dị elu ma n'otu oge ahụ adịghị edebe ụyọkọ a n'oge ibu, na-echekwa ego. Ọfọn, gbakwunyere ihe a niile na-eme na-akpaghị aka na-enweghị ikere òkè gị. The autoscaler n'onwe ya abụghị ihe ọ bụla karịa otu arịrịọ nke onye njikwa ụyọkọ API na onye na-eweta igwe ojii API, nke edere dịka otu ezi uche siri dị. Ihe ị kwesịrị icheta bụ nkewa nke ọnụ ụzọ 3, dịka anyị dere na mbụ. Ị ga-enwekwa obi ụtọ.

isi: www.habr.com

Tinye a comment