Sei bhangi richida maAIOps uye amburera yekutarisa, kana kuti hukama hwevatengi hwakavakirwa papi?

Muzvinyorwa zveHabré, ndakatonyora nezve ruzivo rwangu rwekuvaka hukama nechikwata changu (pano inotaura nezve magadzirirwo echibvumirano chekudyidzana paunotanga bhizinesi idzva kuitira kuti bhizinesi risaparare). Uye zvino ndinoda kutaura nezve nzira yekuvaka hukama nevatengi, sezvo pasina ivo hapazovi nekuparara. Ndinovimba kuti chinyorwa ichi chichabatsira kune vanotanga vari kutanga kutengesa chigadzirwa chavo kumabhizinesi makuru.

Parizvino ndiri kutungamira kunotanga kunonzi MONQ Digital lab, uko ini nechikwata changu tiri kugadzira chigadzirwa chekugadzira otomatiki maitiro ekutsigira uye kushanda kwekambani IT. Kupinda mumusika harisi basa riri nyore uye takatanga nebasa rechikoro diki, takapfuura nenyanzvi dzemusika, vatinoshanda navo uye tikaita chikamu chemusika. Mubvunzo mukuru waive wekunzwisisa "marwadzo aani atingarapa zvakanyanya?"

Mabhangi akaita kuti ive TOP 3 zvikamu. Uye zvechokwadi, yekutanga pane iyo rondedzero yaive Tinkoff uye Sberbank. Patakashanyira nyanzvi dzemusika webhangi, vakati: suma chigadzirwa chako ipapo, uye nzira inoenda kumusika webhangi ichavhurika. Takaedza kupinda imomo nekoko, asi kukundikana kwakatimirira paSberbank, uye vakomana veTinkoff vakave vakasununguka zvikuru kune kukurukurirana kunobudirira nekutanga kweRussia (zvichida nekuda kwekuti Sber panguva iyoyo. buy ingangoita bhiriyoni yevakwikwidzi vedu vekuMadokero). Mumwedzi mumwe chete takatanga chirongwa chekuedza. Zvakaitika sei, verenga.

Tave tichibata nenyaya dzekushanda nekutarisa kwemakore mazhinji, ikozvino tiri kuita chigadzirwa chedu munharaunda yeruzhinji, muinishuwarenzi, mumabhangi, mumakambani enharembozha, kumwe kuitiswa kwaive nendege (purojekiti isati yasvika, isu hatina kana funga kuti kubhururuka nendege yaive indasitiri inotsamira paIT, uye Iye zvino isu tinovimba, kunyangwe COVID, kuti kambani ichabuda yosimuka).

Chigadzirwa chatinogadzira ndechebhizinesi software, iyo AIOps (Artificial Intelligence yeIT Operations, kana ITOps) chikamu. Zvinangwa zvikuru zvekushandisa masisitimu akadai seyero yekukura mukambani inowedzera:

1. Dzima moto: tsvaga kutadza, bvisa rwizi rwekuzivisa kubva kumarara, govera mabasa uye zviitiko kune avo vane mhosva;
2. Wedzera kushanda kwebasa reIT: kuderedza nguva yekugadzirisa zviitiko, kuratidza zvikonzero zvekukundikana, kuwedzera kujekesa kwehutano hweTI;
3. Wedzera kushanda kwebhizinesi: kuderedza huwandu hwebasa remaoko, kuderedza njodzi, kuwedzera kuvimbika kwevatengi.

Muchiitiko chedu, mabhangi ane "marwadzo" anotevera nekutarisa kwakafanana nezvose zvakakura IT zvivakwa:

• "ndiani anoziva": kune akawanda madhipatimendi ehunyanzvi, anenge munhu wese ane kanenge imwe yekutarisa system, uye mazhinji ane anopfuura rimwe;
• "Mosquito swarm" yechenjedzo: yega yega system inoburitsa mazana uye inobhomba vese vane basa navo (dzimwe nguva zvakare pakati pemadhipatimendi). Zvakaoma kugara uchichengetedza tarisiro yekutonga pachiziviso chega chega; kukurumidza kwavo uye kukosha kunoiswa nekuda kwenhamba huru;
• mabhanga makuru - vatungamiriri vechikamu havadi kungoramba vachitarisa masisitimu avo, kuziva pane kukundikana, asiwo mashiripiti chaiwo eAI - kuita kuti masisitimu azvitarise, azvifungidzire uye azvigadzirise.

Patakasvika kumusangano wokutanga paTinkoff, takakurumidza kuudzwa kuti vakanga vasina zvinetso nekuongorora uye hapana chakavarwadza, uye mubvunzo mukuru waiva wokuti: “Chii chatingapa vaya vari kutoita zvakanaka?”

Hurukuro yacho yaive refu, takakurukura kuti ma microservices avo anovakwa sei, madhipatimendi anoshanda sei, izvo zvinetso zvezvivakwa zvinonyanya kunzwisiswa, izvo zvisinganyanyi kunetseka kune vashandisi, aripi "mapofu mapofu", uye ndezvipi zvinangwa zvavo uye SLAs.

Nenzira, maSLA ebhangi anonakidza chaizvo. Semuenzaniso, chiitiko chekutanga chekuwanikwa kwetiweki chingangotora maminetsi mashoma kugadzirisa. Mutengo wekukanganisa uye nguva yekudzikira pano, hongu, inoshamisa.

Nekuda kweizvozvo, takaona nzvimbo dzinoverengeka dzekubatana:

1. nhanho yekutanga ndeyekutarisa amburera kuwedzera kukurumidza kwekugadzirisa chiitiko
2. nhanho yechipiri ndeyekuita otomatiki kudzikisa njodzi uye kuderedza mitengo yekuyera iyo IT department.

"Mavara machena" akati wandei aigona kupendwa nemavara akajeka echenjedzo chete nekugadzirisa ruzivo kubva kune akati wandei ekutarisa masisitimu, sezvo zvanga zvisingaite kutora metrics zvakananga; zvaidiwa zvakare kuisa pakati data kubva kune akasiyana masisitimu ekutarisa kuenda ku "imwe skrini" kuitira kuti. kuti tinzwisise mufananidzo wose wezvaiitika. "Amburera" akakodzera basa iri uye takasangana izvi zvinodiwa ipapo.

Chinhu chakakosha, mumaonero edu, muhukama nevatengi kutendeseka. Mushure mekukurukurirana kwekutanga uye kuverenga kwemutengo werezinesi, zvakanzi sezvo mutengo wakaderera, zvingave zvakakosha kutenga rezinesi ipapo (zvichienzaniswa neDynatrace Klyuch-Astrom kubva kuchinyorwa chiri pamusoro nezve green bank, yedu. rezinesi inodhura kwete chetatu chebhiriyoni, asi zviuru gumi nezviviri zveRubles pamwedzi ye12 gigabyte, yeSber inodhura kakawanda zvakachipa). Asi takakurumidza kuvaudza zvatinazvo nezvatisina. Zvichida mumiririri wekutengesa kubva kumusanganisi mukuru anogona kuti "hongu, tinogona kuita zvose, zvechokwadi kutenga rezinesi yedu," asi takasarudza kuisa makadhi edu ose patafura. Panguva yekutanga, bhokisi redu rakanga risina kubatanidzwa nePrometheus, uye vhezheni itsva ine otomatiki subsystem yakanga yava kuda kuburitswa, asi isu hatisati taendesa kune vatengi.

The pilot project yakatanga, miganhu yayo yakatemwa uye takapiwa 2 months. Mabasa makuru aive:

• gadzirira vhezheni itsva yepuratifomu uye uiise mubhengi rekuvaka
• batanidza 2 yekutarisa masisitimu (Zabbix uye Prometheus);
• tumira zviziviso kune avo vane basa muSlack uye kuburikidza neSMS;
• mhanyisa autohealing scripts.

Mwedzi wekutanga wepurojekiti yekuedza wakapedzwa kugadzirira shanduro itsva yepuratifomu mune super-fast mode yezvinodiwa zvechirongwa chekuyedza. Iyo vhezheni nyowani pakarepo inosanganisira kubatanidzwa nePrometheus uye auto-kuporesa. Tinotenda kuchikwata chedu chebudiriro, havana kurara husiku hwakati wandei, asi vakaburitsa zvavakavimbisa vasina kupotsa mazuva ekupedzisira ezvimwe zvisungo zvakamboitwa.

Patakanga tichigadzira mutyairi, takasangana nedambudziko idzva raigona kuvhara chirongwa pamberi pehurongwa: kutumira chenjedzo kune vatumwa nekukurumidza uye kuburikidza neSMS, taida chinouya uye chinobuda chinongedzo kumaseva eMicrosoft Azure (panguva iyoyo taishandisa chikuva ichi. kutumira zviziviso kuSlack) uye yekunze yekutumira sevhisi SMS. Asi muchirongwa ichi, kuchengetedzwa kwainyanya kutariswa. Maererano nemutemo webhangi, "maburi" akadaro aisagona kuvhurwa mumamiriro ezvinhu chero api zvawo. Zvese zvaifanira kushanda kubva pakavharika loop. Isu takapihwa kushandisa API yedu yemukati masevhisi anotumira chenjedzo kuSlack uye kuburikidza neSMS, asi isu hatina kuwana mukana wekubatanidza masevhisi akadaro kunze kwebhokisi.

Manheru ekukakavadzana neboka rekusimudzira rakapera nekubudirira kutsvaga mhinduro. Mushure mekuongorora nekumashure, takawana basa rimwe chete ratasina kumbova nenguva yakakwana uye yakakosha - kugadzira plug-in system kuitira kuti zvikwata zvekushandisa kana mutengi akwanise kunyora ekuwedzera, achiwedzera kugona kwepuratifomu.

Asi isu taive nemwedzi chaiwo wasara, panguva yataifanira kuisa zvese, kugadzirisa uye kuendesa otomatiki.

Sekureva kwaSergei, mugadziri wedu mukuru, zvinotora kanenge mwedzi kuita plug-in system.

Takanga tisina nguva...

Paive nemhinduro imwe chete - enda kumutengi uye utaure zvese sezvazviri. Kurukurai nguva yekupedzisira pamwe chete. Uye zvakashanda. Takapiwa mamwe 2 weeks. Vaivewo nemazuva avo ekupedzisira uye zvisungo zvemukati zvekuratidza mibairo, asi vaive nemavhiki maviri ekuchengetedza. Pakupedzisira, tinoisa zvinhu zvose pamutsara. Zvaisaita kukanganisa. Kutendeseka uye nzira yekudyidzana yakabhadhara zvakare.

Nekuda kwemutyairi, akati wandei akakosha ehunyanzvi mhedzisiro uye mhedziso dzakawanikwa:

Takaedza mashandiro matsva ekugadzirisa zviziviso

Iyo yakatumirwa sisitimu yakatanga kugashira yambiro kubva kuna Prometheus uye nemapoka. Nyevero padambudziko kubva kumutengi wePrometheus aibhururuka masekondi makumi matatu ega ega (kuronga nenguva hakugoneke), uye isu tainetseka kuti zvingaite here kuvabatanidza mu "amburera" pachayo. Zvakazoitika kuti zvinogoneka - kumisikidza kugadziriswa kwezviziviso mupuratifomu kunoitwa ne script. Izvi zvinoita kuti zvikwanise kuita chero chero logic yekuzvigadzirisa. Isu takatoita zvakajairwa logic papuratifomu nenzira yematemplate - kana iwe usingade kuunza chimwe chinhu chako, unogona kushandisa yakagadzirira-yakagadzirwa.

"Synthetic trigger" interface. Kumisikidza kugadziridzwa kwezviyeuchidzo kubva kune akabatana ekutarisa masisitimu

Yakagadzira mamiriro e "hutano" hwehurongwa

Zvichienderana nechenjedzo, zviitiko zvekutarisa zvakagadzirwa izvo zvakakanganisa hutano hwemagadzirirwo mayuniti (CUs). Tiri kushandisa resource-service model (RSM), iyo inogona kushandisa CMDB yemukati kana kubatanidza yekunze - panguva yechirongwa chekuyedza mutengi haana kubatanidza CMDB yake.

Interface yekushanda neiyo resource-service modhi. Mutyairi weRSM.

Zvakanaka, muchokwadi, mutengi anozopedzisira aine imwe yekutarisa skrini, uko zviitiko kubva kune akasiyana masisitimu zvinoonekwa. Parizvino, masisitimu maviri akabatana ne "amburera" - Zabbix uye Prometheus, uye yemukati yekutarisa sisitimu yepuratifomu pachayo.

Analytics interface. Single monitoring screen.

Yakatangisa process automation

Zviitiko zvekuona zvakakonzera kutangwa kwezviito zvakafanorongwa - kutumira chenjedzo, kumhanya zvinyorwa, kunyoresa / kupfumisa zviitiko - iyo yekupedzisira haina kuyedzwa nemutengi uyu, nekuti. muchirongwa chekuedza pakanga pasina kubatanidzwa nedhesiki rebasa.

Action zvigadziriso interface. Tumira zviziviso kuSlack uye tangazve sevha.

Yakawedzera chigadzirwa kushanda

Pakukurukura otomatiki scripts, mutengi akakumbira bash rutsigiro uye interface umo zvinyorwa izvi zvinogona kugadziridzwa zviri nyore. Iyo vhezheni nyowani yakaita zvimwe zvishoma (kugona kunyora yakazara-yakazara inonzwisisika inovaka muLua nerutsigiro rwecURL, SSH uye SNMP) uye yakaitwa mashandiro ayo anobvumidza iwe kutonga kutenderera kwehupenyu hwechinyorwa (gadzira, gadzirisa, kutonga kweshanduro. , bvisa uye chengetedza).

Interface yekushanda ne autohealing scripts. Server reboot script kuburikidza neSSH.

Main mhedziso

Munguva yemutyairi, nyaya dzemushandisi dzakagadzirwawo dzinovandudza mashandiro azvino uye kuwedzera kukosha kwemutengi, heano mamwe acho:

• shandisa kugona kuendesa vhezheni zvakananga kubva kune yambiro kuenda kune autohealing script;
• wedzera mvumo kupuratifomu kuburikidza neActive Directory.

Uye isu takagamuchira mamwe matambudziko epasirese - "kuvaka" chigadzirwa nemamwe maitiro:

• kuvakwa otomatiki kwechishandiso-sevhisi modhi yakavakirwa paML, pane mitemo nevamiriri (zvichida dambudziko guru ikozvino);
• rutsigiro rwekuwedzera scripting uye logic mitauro (uye iyi ichave JavaScript).

Mukufunga kwangu chinonyanya kukoshaZvinoratidzwa nemutyairi uyu zvinhu zviviri:

1. Kudyidzana nemutengi ndiyo kiyi yekubudirira, kana kutaurirana kunoshanda kwakavakirwa pahwaro hwekutendeseka uye kuvhurika, uye mutengi anova chikamu chechikwata chinowana mhedzisiro yakakosha munguva pfupi.
2. Pasina mamiriro ezvinhu zvakakosha "kugadzirisa" uye kuvaka "madondoro" - mhinduro dzehurongwa chete. Zviri nani kushandisa imwe nguva shoma, asi ita sarudzo yekugadzirisa iyo ichashandiswa nevamwe vatengi. Nenzira, izvi ndizvo zvakaitika, iyo plugin system uye kubviswa kwekutsamira paAzure kwakapa humwe kukosha kune vamwe vatengi (hesi, Federal Law 152).

Source: www.habr.com