Kungani ibhange lidinga ama-AIOps nokuqapha isambulela, noma busekelwe kuphi ubudlelwano bamakhasimende?

Ezincwadini eziphathelene no-Habré, ngase ngivele ngibhale mayelana nolwazi lwami lokwakha ubudlelwano nethimba lami (lapha ikhuluma ngokuthi senziwa kanjani isivumelwano sokusebenzisana lapho uqala ibhizinisi elisha ukuze ibhizinisi lingabhilidi). Futhi manje ngithanda ukukhuluma ngendlela yokwakha ubudlelwano namakhasimende, ngoba ngaphandle kwabo ngeke kube khona ukuhlukana. Ngithemba ukuthi lesi sihloko sizoba usizo kwabaqalayo abaqala ukuthengisa umkhiqizo wabo emabhizinisini amakhulu.

Okwamanje ngiphokophele esiqalisweni esibizwa nge-MONQ Digital lab, lapho mina nethimba lami sakha umkhiqizo wokuzenzakalela izinqubo zokusekela nokusebenza kwe-IT yebhizinisi. Ukungena emakethe akuwona umsebenzi olula futhi saqala ngomsebenzi wesikole omncane, sadlula kochwepheshe bemakethe, ozakwethu futhi senza ukuhlukaniswa kwemakethe. Umbuzo omkhulu bekuwukuqonda ukuthi “obani izinhlungu esingaselapha kangcono?”

Amabhange afinyelele kumasegmenti angu-TOP 3. Futhi-ke, abokuqala ohlwini kwaba Tinkoff futhi Sberbank. Lapho sivakashela ochwepheshe bemakethe yasebhange, bathi: ngenisa umkhiqizo wakho lapho, futhi indlela eya emakethe yasebhange izovuleka. Sazama ukungena lapho nalaphaya, kodwa ukwehluleka kusilindele e-Sberbank, futhi abafana baseTinkoff bavuleleka kakhulu ekukhulumisaneni okuphumelelayo nabaqaphi baseRussia (mhlawumbe ngenxa yokuthi i-Sber ngaleso sikhathi uthenge cishe isigidigidi sezimbangi zethu zaseNtshonalanga). Ngenyanga eyodwa saqala umsebenzi wokuhlola. Kwenzeka kanjani, qhubeka.

Sekuyiminyaka eminingi sibhekene nezinkinga zokusebenza nokuqapha, manje sisebenzisa umkhiqizo wethu emkhakheni kahulumeni, kumshwalense, emabhange, ezinkampanini ze-telecom, ukuqaliswa okukodwa bekunenkampani yezindiza (ngaphambi kwephrojekthi, asizange cabanga ukuthi izindiza bekuyimboni encike ku-IT, futhi Manje siyethemba ngempela, naphezu kwe-COVID, ukuthi inkampani izovela futhi ihambe).

Umkhiqizo esiwenzayo ungowesofthiwe yebhizinisi, ingxenye ye-AIOps (Artificial Intelligence for IT Operations, noma i-ITOps). Izinjongo eziyinhloko zokuqalisa izinhlelo ezifana nezinga lokuvuthwa kwenqubo enkampanini likhuphuka:

1. Cisha imililo: khomba ukwehluleka, sula umfudlana wezixwayiso emfucumfucwini, yabela labo abanomthwalo wemfanelo imisebenzi kanye nezigameko;
2. Ukwandisa ukusebenza kahle kwenkonzo ye-IT: ukunciphisa isikhathi sokuxazulula izigameko, bonisa izimbangela zokuhluleka, ukwandisa ukucaca kwesimo se-IT;
3. Khulisa ukusebenza kahle kwebhizinisi: nciphisa inani lemisebenzi yezandla, unciphise ubungozi, ukhulise ukwethembeka kwamakhasimende.

Ngokuhlangenwe nakho kwethu, amabhange analezi "zinhlungu" ezilandelayo zokuqapha okufanayo nazo zonke izingqalasizinda ezinkulu ze-IT:

• “ngubani owaziyo”: kuneminyango eminingi yezobuchwepheshe, cishe wonke umuntu unesistimu yokuqapha okungenani eyodwa, futhi eminingi inengaphezu kowodwa;
• "iswebezane lomiyane" lezaziso: isistimu ngayinye ikhiqiza amakhulukhulu futhi iqhume ngamabhomu bonke abathintekayo (ngezinye izikhathi futhi phakathi kweminyango). Kunzima ukugcina ukugxila kokulawula kwisaziso ngasinye, ukuphuthuma nokubaluleka kwazo kulinganiswa ngenxa yenani elikhulu;
• amabhange amakhulu - abaholi bomkhakha abafuni nje kuphela ukuqapha ngokuqhubekayo izinhlelo zabo, ukwazi lapho kukhona ukwehluleka, kodwa futhi umlingo wangempela we-AI - ukwenza izinhlelo zizihlole, zizibikezele futhi zizilungise.

Lapho sifika emhlanganweni wokuqala eTinkoff, ngokushesha satshelwa ukuthi babengenayo inkinga yokuqapha futhi akukho lutho olubalimazayo, futhi umbuzo oyinhloko wawuthi: “Yini esingabanikeza yona labo abenza kahle kakade?”

Ingxoxo yayiyinde, sixoxe ngokuthi ama-microservices abo akhiwa kanjani, ukuthi iminyango isebenza kanjani, yiziphi izinkinga zengqalasizinda ezibucayi kakhulu, ezingazweli kakhulu kubasebenzisi, ziphi “izindawo eziyizimpumputhe”, futhi iyini imigomo nama-SLA.

Ngendlela, ama-SLA ebhange ahlaba umxhwele ngempela. Isibonelo, isigameko sokutholakala kwenethiwekhi esingu-XNUMX esibalulekile singase sithathe amaminithi ambalwa kuphela ukuxazulula. Izindleko zephutha nesikhathi sokuphumula lapha, yiqiniso, ziyamangalisa.

Ngenxa yalokho, sihlonze izindawo ezimbalwa zokubambisana:

1. isigaba sokuqala ukuqapha isambulela ukwandisa isivinini sokuxazulula isigameko
2. isigaba sesibili i-automation yenqubo yokunciphisa ubungozi kanye nokunciphisa izindleko zokukala umnyango we-IT.

“Amabala amhlophe” amaningana ayengapendwe ngemibala egqamile yezaziso kuphela ngokucubungula ulwazi oluvela ezinhlelweni zokuqapha eziningana, njengoba kwakungenakwenzeka ukuthatha amamethrikhi ngokuqondile; kwakudingeka futhi ukuthi idatha yezinhlelo zokuqapha ibekwe endaweni eyodwa iye “esikrinini esisodwa” ukuze ukuqonda isithombe sonke salokho okwakwenzeka. “Izambulela” ziwufanele lo msebenzi futhi sahlangabezana nalezi zidingo ngaleso sikhathi.

Into ebaluleke kakhulu, ngokombono wethu, ebudlelwaneni namaklayenti ukwethembeka. Ngemva kwengxoxo yokuqala nokubalwa kwezindleko zelayisensi, kwathiwa njengoba izindleko ziphansi kakhulu, kungase kudingeke ukuthenga ilayisense ngokushesha (uma kuqhathaniswa noDynatrace Klyuch-Astrom kusukela esihlokweni esingenhla mayelana nebhange eliluhlaza, yethu Ilayisensi ayibizi ingxenye yesithathu yebhiliyoni, kodwa ama-ruble ayizinkulungwane ezingu-12 ngenyanga ngegigabhayithi engu-1, ku-Sber izobiza izikhathi ezimbalwa ezishibhile). Kodwa ngokushesha sabatshela lokho esinakho nesingenakho. Mhlawumbe omele ukuthengisa ovela kumdidiyeli omkhulu angathi “yebo, singenza konke, vele sithenge ilayisense yethu,” kodwa sanquma ukubeka wonke amakhadi ethu etafuleni. Ngesikhathi sokwethulwa, ibhokisi lethu belingenalo ukuhlanganiswa ne-Prometheus, futhi inguqulo entsha ene-automation subsystem yayisizokhishwa, kodwa asikakayithumeli kumakhasimende okwamanje.

Iphrojekthi yokuhlola yaqala, kwanqunywa imingcele yayo futhi sanikezwa izinyanga ezi-2. Imisebenzi esemqoka yayiyi:

• lungisa inguqulo entsha yesikhulumi futhi uyisebenzise kungqalasizinda yebhange
• xhuma izinhlelo zokuqapha ezi-2 (i-Zabbix ne-Prometheus);
• thumela izaziso kulabo abanomthwalo wemfanelo ku-Slack nange-SMS;
• sebenzisa imibhalo ye-autohealing.

Inyanga yokuqala yephrojekthi yokuhlola yachithwa kulungiselelwa inguqulo entsha yesikhulumi ngemodi eshesha kakhulu yezidingo zephrojekthi yokuhlola. Inguqulo entsha ihlanganisa ngokushesha ukuhlanganiswa ne-Prometheus kanye nokuphulukisa okuzenzakalelayo. Sibonga ithimba lethu labathuthukisi, abazange balale ubusuku obuthile, kodwa bakhiphe ababekuthembisile ngaphandle kokuphuthelwa yiminqamulajuqu yezinye izibopho ezenziwa ngaphambilini.

Ngenkathi simisa umshayeli wendiza, sihlangabezane nenkinga entsha engavala iphrojekthi ngaphambi kwesikhathi esimisiwe: ukuthumela izexwayiso kuzithunywa ezisheshayo nange-SMS, sasidinga ukuxhumana okungenayo nokuphumayo kumaseva e-Microsoft Azure (ngaleso sikhathi sasisebenzisa le nkundla. ukuthumela izexwayiso ku-Slack) kanye ne-SMS yesevisi yokuthumela yangaphandle. Kodwa kule phrojekthi, ukuphepha bekugxilwe kakhulu. Ngokuvumelana nenqubomgomo yebhange, “izimbobo” ezinjalo azikwazanga ukuvulwa ngaphansi kwanoma yiziphi izimo. Konke kwakufanele kusebenze ku-loop evaliwe. Sanikezwa ukusebenzisa i-API yezinsizakalo zethu zangaphakathi ezithumela izexwayiso ku-Slack nange-SMS, kodwa asizange sibe nethuba lokuxhuma lezi zinsizakalo ngaphandle kwebhokisi.

Ubusuku benkulumompikiswano nethimba lezokuthuthukiswa kwaphela ngokusesha okuyimpumelelo kwesixazululo. Ngemva kokuphenya ukusilela emuva, sithole umsebenzi owodwa esingakaze sibe nesikhathi esanele nokubalulekile kuwo - ukwakha isistimu ye-plug-in ukuze amaqembu okusebenzisa noma iklayenti likwazi ukubhala izengezo ngokwalo, sandise amakhono enkundla.

Kodwa besinenyanga esele, lapho bekufanele sifake yonke into, silungiselele futhi sisebenzise i-automation.

Ngokusho kukaSergei, umakhi wethu omkhulu, kuthatha okungenani inyanga ukusebenzisa uhlelo lwe-plug-in.

Besingenasikhathi...

Kwakunesixazululo esisodwa kuphela - vakashela iklayenti futhi utshele yonke into njengoba injalo. Xoxani ngokushintsha komnqamulajuqu ndawonye. Futhi kwasebenza. Sanikezwa amaviki ama-2 engeziwe. Futhi babenezikhathi zabo zokugcina kanye nezibopho zangaphakathi zokukhombisa imiphumela, kodwa babe namaviki angu-2 okugodla. Ekugcineni, sibeka yonke into emgqeni. Kwakungenakwenzeka ukumosha. Ukwethembeka nendlela yokusebenzisana kuphinde kwaba nezithelo.

Njengomphumela wokuhlola, kutholwe imiphumela embalwa ebalulekile yezobuchwepheshe neziphetho:

Sihlole umsebenzi omusha wokucubungula izexwayiso

Uhlelo olusetshenzisiwe lwaqala ukuthola izexwayiso ngendlela efanele ku-Prometheus futhi luziqoqe. Izaziso ezimayelana nenkinga ezivela kuklayenti le-Prometheus bezindiza njalo ngemizuzwana engama-30 (ukuqoqa ngesikhathi akuvunyelwe), futhi besizibuza ukuthi kungenzeka yini ukuzihlanganisa “kusambulela” ngokwaso. Kuvele ukuthi kungenzeka - ukusetha ukucutshungulwa kwezixwayiso endaweni yesikhulumi kusetshenziswa iskripthi. Lokhu kwenza kube nokwenzeka ukusebenzisa cishe noma iyiphi i-logic yokuwacubungula. Sesivele sisebenzise ukucabanga okujwayelekile endaweni yesikhulumi ngesimo sezifanekiso - uma ungafuni ukuza nokuthile okungokwakho, ungasebenzisa esenziwe ngomumo.

"Synthetic trigger" interface. Ukusetha ukucutshungulwa kwezixwayiso ezivela kumasistimu okuqapha axhunyiwe

Yakha isimo "sezempilo" sohlelo

Ngokusekelwe kuzixwayiso, izehlakalo zokuqapha zadalwa ezithinte impilo yamayunithi okulungiselela (CUs). Sisebenzisa imodeli yesevisi yensiza (RSM), engasebenzisa i-CMDB yangaphakathi noma ixhume yangaphandle - phakathi nephrojekthi yokuhlola iklayenti alizange lixhume i-CMDB yalo.

Isixhumi esibonakalayo sokusebenza ngemodeli yensiza-yensiza. Umshayeli we-RSM.

Eqinisweni, iklayenti ekugcineni linesikrini esisodwa sokuqapha, lapho izenzakalo ezivela ezinhlelweni ezahlukene zibonakala. Njengamanje, izinhlelo ezimbili zixhunywe "kwisambulela" - i-Zabbix ne-Prometheus, kanye nesistimu yokuqapha yangaphakathi yesikhulumi ngokwayo.

Isixhumi esibonakalayo sezibalo. Isikrini sokuqapha esisodwa.

Kwethulwe inqubo ezenzakalelayo

Izehlakalo zokuqapha zicuphe ukwethulwa kwezenzo ezilungiselelwe kusengaphambili - ukuthumela izexwayiso, ukusebenzisa imibhalo, ukubhalisa/ukucebisa izehlakalo - lokhu akuzange kuzanywe ngaleli klayenti, ngoba kuphrojekthi yokuhlola kwakungekho ukuhlanganiswa nedeski lesevisi.

Isixhumanisi sezilungiselelo zesenzo. Thumela izexwayiso ku-Slack futhi uqalise kabusha iseva.

Ukusebenza komkhiqizo okunwetshiwe

Lapho kudingidwa izikripthi zokuzenzakalela, iklayenti licele usekelo lwe-bash kanye nesixhumi esibonakalayo lapho le mibhalo ingacushwa kalula. Inguqulo entsha yenze okwengeziwe (ikhono lokubhala izakhiwo ezinengqondo ezigcwele e-Lua ngokusekelwa kwe-cURL, i-SSH ne-SNMP) kanye nokusebenza okusebenzayo okukuvumela ukuthi ulawule umjikelezo wempilo weskripthi (dala, hlela, ukulawula inguqulo. , susa bese ufaka kungobo yomlando).

Isixhumi esibonakalayo sokusebenza ngemibhalo yokupholisa ngokuzenzakalela. Iseva iqalisa kabusha iskripthi nge-SSH.

Ukutholwa Okusemqoka

Ngesikhathi sokuhlola, izindaba zabasebenzisi nazo zadalwa ezithuthukisa ukusebenza kwamanje futhi zikhuphule inani leklayenti, nazi ezinye zazo:

• sebenzisa ikhono lokudlulisa okuhlukile ngokuqondile kusuka kusixwayiso kuya kusikripthi sokuphulukisa ngokuzenzakalela;
• engeza ukugunyazwa endaweni ngohlelo Lwemibhalo Olusebenzayo.

Futhi sithole izinselele eziningi zomhlaba - "ukwakha" umkhiqizo ngamanye amakhono:

• ukwakhiwa okuzenzakalelayo kwemodeli yesevisi yensiza esekelwe ku-ML, kunemithetho nama-ejenti (cishe inselelo enkulu manje);
• ukusekelwa kwezilimi ezengeziwe zokubhala nezinengqondo (futhi lokhu kuzoba yiJavaScript).

Ngokubona kwami okubaluleke kakhuluOkuboniswa yilo mshayeli yizinto ezimbili:

1. Ukubambisana neklayenti kuyisihluthulelo sokusebenza kahle, lapho ukuxhumana okuphumelelayo kwakhiwe ngesisekelo sokwethembeka nokuvuleleka, futhi iklayenti liba yingxenye yeqembu elizuza imiphumela ebalulekile ngesikhathi esifushane.
2. Ngaphansi kwezimo akudingekile "ukwenza ngokwezifiso" futhi wakhe "izinduku" - izixazululo zesistimu kuphela. Kungcono ukuchitha isikhathi esincane, kodwa yenza isisombululo sesistimu esizosetshenziswa amanye amaklayenti. Ngendlela, yilokho okwenzekile, uhlelo lwe-plugin kanye nokuqedwa kokuncika ku-Azure kunikeze inani elingeziwe kwamanye amaklayenti (sawubona, Federal Law 152).

Source: www.habr.com