Chifukwa chiyani banki ikufunika ma AIOps ndi kuyang'anira maambulera, kapena maubwenzi a kasitomala amatengera chiyani?

M'mabuku a Habré, ndidalemba kale za zomwe ndakumana nazo popanga mgwirizano ndi gulu langa (apa amakamba za momwe angapangire mgwirizano waubwenzi poyambitsa bizinesi yatsopano kuti bizinesi isagwere). Ndipo tsopano ndikufuna kulankhula za momwe mungamangire mgwirizano ndi makasitomala, popeza popanda iwo sipadzakhalanso kanthu. Ndikukhulupirira kuti nkhaniyi idzakhala yothandiza kwa oyambitsa omwe akuyamba kugulitsa malonda awo ku malonda akuluakulu.

Panopa ndikuyambitsa zoyamba zotchedwa MONQ Digital lab, komwe ine ndi gulu langa tikupanga chinthu chothandizira ndi kuyendetsa makampani a IT. Kulowa mumsika si ntchito yophweka ndipo tinayamba ndi homuweki yaing'ono, tinadutsa akatswiri a msika, okondedwa athu ndikuchita magawo a msika. Funso lalikulu linali kumvetsetsa "zowawa za ndani tingachiritse bwino?"

Mabanki adapanga gawo la TOP 3. Ndipo ndithudi, woyamba pa mndandanda anali Tinkoff ndi Sberbank. Titayendera akatswiri amsika amabanki, adati: wonetsani malonda anu kumeneko, ndipo njira yopita kumisika yamabanki idzatsegulidwa. Tinayesetsa kulowa apo ndi apo, koma kulephera kunatiyembekezera ku Sberbank, ndipo anyamata ochokera ku Tinkoff adakhala omasuka kwambiri kuti azitha kulankhulana bwino ndi oyambitsa aku Russia (mwina chifukwa chakuti Sber panthawiyo anali omasuka kwambiri. anagula pafupifupi biliyoni ya opikisana athu akumadzulo). Patangotha ​​mwezi umodzi tinayamba ntchito yoyeserera. Momwe zidachitikira, werengani.

Takhala tikukumana ndi zovuta zogwirira ntchito ndi kuyang'anira kwa zaka zambiri, tsopano tikugwiritsa ntchito malonda athu m'maboma, mu inshuwaransi, m'mabanki, m'makampani a telecom, kukhazikitsidwa kumodzi kunali ndi ndege (ntchito isanayambe, sitinachitepo kanthu. ndikuganiza kuti ndegeyo inali bizinesi yodalira IT, ndipo Tsopano tikukhulupirira, ngakhale COVID, kampaniyo ituluka ndikunyamuka).

Zomwe timapanga ndi zamapulogalamu amakampani, gawo la AIOps (Artificial Intelligence for IT Operations, kapena ITOps). Zolinga zazikulu zogwiritsira ntchito machitidwe monga kukula kwa ndondomeko mu kampani kumawonjezeka:

1. Zimitsani moto: zindikirani zolephera, chotsani zidziwitso kuchokera ku zinyalala, perekani ntchito ndi zochitika kwa omwe ali ndi udindo;
2. Wonjezerani mphamvu ya ntchito ya IT: kuchepetsa nthawi yothetsera zochitika, kusonyeza zomwe zimayambitsa zolephera, kuwonjezera kuwonekera kwa chikhalidwe cha IT;
3. Wonjezerani mphamvu zamabizinesi: kuchepetsa kuchuluka kwa ntchito zamanja, kuchepetsa zoopsa, kuwonjezera kukhulupirika kwa makasitomala.

Zomwe takumana nazo, mabanki ali ndi "zowawa" zotsatirazi ndikuwunika kofanana ndi zida zonse zazikulu za IT:

• "Ndani akudziwa chiyani": pali madipatimenti ambiri aukadaulo, pafupifupi aliyense ali ndi njira imodzi yowunikira, ndipo ambiri amakhala ndi opitilira imodzi;
• chenjezo la "udzudzu": dongosolo lililonse limapanga mazana ndikuwombera onse omwe ali nawo (nthawi zina pakati pa madipatimenti). Ndikovuta kusungabe kuyang'ana kwaulamuliro pachidziwitso chilichonse; kufulumira kwawo ndi kufunikira kwawo kumayendetsedwa chifukwa cha kuchuluka;
• mabanki akuluakulu - atsogoleri amagulu safuna kuti azingoyang'anira mosalekeza machitidwe awo, kuti adziwe komwe kuli zolephera, komanso matsenga enieni a AI - kupanga machitidwe odziwonetsera okha, kudziwonetsera okha komanso kudziwongolera okha.

Titafika ku msonkhano woyamba ku Tinkoff, tinauzidwa mwamsanga kuti analibe vuto ndi kuyang’anira ndipo palibe chimene chinawapweteka, ndipo funso lalikulu linali lakuti: “Kodi tingawapatse chiyani iwo amene akuchita bwino kale?”

Kukambitsirana kunali kwautali, tidakambirana momwe ma microservices awo amamangidwira, momwe madipatimenti amagwirira ntchito, zomwe zovuta zowonongeka zimakhala zovuta kwambiri, zomwe sizimakhudzidwa kwambiri ndi ogwiritsa ntchito, "malo osawona" ali kuti, ndi zolinga zawo ndi SLAs.

Mwa njira, ma SLA a banki ndi ochititsa chidwi kwambiri. Mwachitsanzo, vuto loyamba la kupezeka kwa netiweki lingatenge mphindi zochepa kuti lithetse. Mtengo wa zolakwika ndi nthawi yopuma pano, ndithudi, ndizochititsa chidwi.

Zotsatira zake, tazindikira magawo angapo a mgwirizano:

1. gawo loyamba ndikuyang'anira maambulera kuti muwonjezere liwiro la kuthetsa zochitika
2. gawo lachiwiri ndi makina odzipangira okha kuti achepetse zoopsa komanso kuchepetsa ndalama zokulitsira dipatimenti ya IT.

"Mawanga oyera" angapo amatha kujambulidwa mumitundu yowala ya zidziwitso pokhapokha pokonza zidziwitso kuchokera ku machitidwe angapo owunikira, popeza kunali kosatheka kutenga ma metrics mwachindunji; kunali kofunikiranso kuyika deta pakati pa machitidwe osiyanasiyana owunikira pa "screen imodzi" kuti kuti amvetse chithunzi chonse cha zimene zinali kuchitika. "Maambulera" ndi oyenera ntchitoyi ndipo tinakwaniritsa zofunikira izi.

Chinthu chofunika kwambiri, m'malingaliro athu, mu maubwenzi ndi makasitomala ndi kukhulupirika. Pambuyo pokambirana koyamba ndikuwerengera mtengo wa chilolezocho, zidanenedwa kuti popeza mtengo wake ndi wotsika kwambiri, ungakhale woyenera kugula chilolezo nthawi yomweyo (poyerekeza ndi Dynatrace Klyuch-Astrom kuchokera m'nkhani yomwe ili pamwambapa yokhudza banki yobiriwira, yathu. layisensi sichimawononga gawo limodzi mwa magawo atatu a biliyoni, koma ma ruble 12 pamwezi pa gigabyte 1, kwa Sber imayenera kutsika mtengo kangapo). Koma nthawi yomweyo tinawauza zomwe tili nazo komanso zomwe tilibe. Mwinamwake wogulitsa malonda kuchokera kwa ophatikizana wamkulu anganene kuti "inde, tikhoza kuchita chirichonse, ndithudi kugula layisensi yathu," koma tinaganiza zoyika makadi athu onse patebulo. Pa nthawi yotsegulira, bokosi lathu linalibe kusakanikirana ndi Prometheus, ndipo mtundu watsopano wokhala ndi makina opangira makina unali pafupi kutulutsidwa, koma sitinatumize kwa makasitomala.

Ntchito yoyeserera idayamba, malire ake adatsimikizika ndipo tidapatsidwa miyezi iwiri. Ntchito zazikuluzikulu zinali:

• konzani mtundu watsopano wa nsanja ndikuyiyika muzomangamanga za banki
• gwirizanitsani machitidwe owunikira a 2 (Zabbix ndi Prometheus);
• tumizani zidziwitso kwa omwe ali ndi udindo ku Slack komanso kudzera pa SMS;
• yendetsani autohealing scripts.

Mwezi woyamba wa polojekiti yoyendetsa ndegeyo unathera pokonzekera mtundu watsopano wa nsanja mumayendedwe othamanga kwambiri pazosowa za polojekiti yoyendetsa. Mtundu watsopano nthawi yomweyo umaphatikizana ndi Prometheus ndi machiritso amoto. Chifukwa cha gulu lathu lachitukuko, sanagone kwa mausiku angapo, koma adatulutsa zomwe adalonjeza popanda kuphonya masiku omaliza a zomwe adalonjeza kale.

Pamene tikukhazikitsa woyendetsa ndegeyo, tinakumana ndi vuto latsopano lomwe lingathe kutseka ntchitoyi nthawi isanakwane: kutumiza zidziwitso kwa amithenga apompopompo komanso kudzera pa SMS, tinkafunikira maulumikizidwe obwera ndi otuluka ku ma seva a Microsoft Azure (panthawiyo tidagwiritsa ntchito nsanja iyi). kutumiza zidziwitso ku Slack) ndi SMS yotumiza kunja. Koma mu ntchitoyi, chitetezo chinali chofunikira kwambiri. Mogwirizana ndi ndondomeko ya banki, "mabowo" oterowo sakanakhoza kutsegulidwa muzochitika zilizonse. Chilichonse chinayenera kugwira ntchito kuchokera ku njira yotseka. Tinapatsidwa kugwiritsa ntchito API ya ntchito zathu zamkati zomwe zimatumiza zidziwitso ku Slack komanso kudzera pa SMS, koma tinalibe mwayi wolumikiza mautumikiwa kunja kwa bokosi.

Madzulo otsutsana ndi gulu lachitukuko anatha ndi kufufuza bwino kwa yankho. Titafufuza zotsalirazo, tapeza ntchito imodzi yomwe sitinakhalepo ndi nthawi yokwanira komanso yofunika kwambiri - kupanga mapulagini kuti magulu ogwiritsira ntchito kapena kasitomala azitha kulemba zowonjezera, kukulitsa luso la nsanja.

Koma tinali ndi mwezi umodzi wokha womwe udatsala, pomwe tidayenera kukhazikitsa chilichonse, kukonza ndikuyika makina opangira okha.

Malinga ndi Sergei, mmisiri wathu wamkulu, zimatengera mwezi umodzi kuti akwaniritse dongosolo la pulagi.

Tinalibe nthawi...

Panali yankho limodzi lokha - pitani kwa kasitomala ndikuwuza zonse momwe zilili. Kambiranani nthawi yomaliza pamodzi. Ndipo zinathandiza. Tinapatsidwa masabata awiri owonjezera. Analinso ndi nthawi yawoyawo komanso zomwe amafunikira kuti awonetse zotsatira, koma anali ndi masabata awiri osungira. Pamapeto pake, timayika zonse pamzere. Zinali zosatheka kusokoneza. Kuona mtima ndi njira ya mgwirizano inapindulanso.

Chifukwa cha woyendetsa, zotsatira zingapo zofunika zaukadaulo zidapezedwa:

Tinayesa magwiridwe antchito atsopano pokonza zidziwitso

Dongosolo lomwe adatumizidwa lidayamba kulandira zochenjeza kuchokera kwa Prometheus ndikuzigawa. Zidziwitso pavuto lochokera kwa kasitomala wa Prometheus anali kuwuluka masekondi aliwonse a 30 (kuphatikiza ndi nthawi sikunatheke), ndipo tinali kudabwa ngati zingatheke kuwaphatikiza mu "ambulera" yokha. Zinapezeka kuti ndizotheka - kukhazikitsa kukonzedwa kwa zidziwitso papulatifomu kumayendetsedwa ndi script. Izi zimapangitsa kuti zitheke kugwiritsa ntchito pafupifupi malingaliro aliwonse powakonza. Takhazikitsa kale malingaliro okhazikika papulatifomu ngati ma templates - ngati simukufuna kukhala ndi china chake, mutha kugwiritsa ntchito chokonzekera.

"Synthetic trigger" mawonekedwe. Kukhazikitsa makonzedwe a zidziwitso kuchokera ku machitidwe owunikira olumikizidwa

Anapanga chikhalidwe cha "thanzi" la dongosolo

Kutengera zidziwitso, zochitika zowunikira zidapangidwa zomwe zidakhudza thanzi la ma configuration units (CUs). Tikugwiritsa ntchito mtundu wa resource-service (RSM), womwe utha kugwiritsa ntchito CMDB yamkati kapena kulumikiza yakunja - panthawi yoyeserera kasitomala sanalumikize CMDB yakeyake.

Chiyankhulo chogwira ntchito ndi chitsanzo cha zothandizira zothandizira. Woyendetsa RSM.

Chabwino, kwenikweni, kasitomala pamapeto pake ali ndi chophimba chimodzi chowunikira, pomwe zochitika zochokera kumachitidwe osiyanasiyana zimawonekera. Pakalipano, machitidwe awiri akugwirizanitsidwa ndi "ambulera" - Zabbix ndi Prometheus, ndi dongosolo loyang'anira mkati mwa nsanja yokha.

Ma analytics mawonekedwe. Single monitoring screen.

Anayambitsa ndondomeko automation

Zochitika zowunikira zidayambitsa kukhazikitsidwa kwa zomwe zidakonzedweratu - kutumiza zidziwitso, kugwiritsa ntchito zolemba, kulembetsa / kulemeretsa zochitika - zomalizazi sizinayesedwe ndi kasitomala uyu, chifukwa. mu ntchito yoyeserera panalibe kuphatikiza ndi desiki lautumiki.

Zosintha zochita mawonekedwe. Tumizani zidziwitso ku Slack ndikuyambitsanso seva.

Ntchito yowonjezera yazinthu

Pokambilana zolemba zokha, kasitomala amapempha thandizo la bash ndi mawonekedwe momwe zolemberazi zitha kukhazikitsidwa mosavuta. Mtundu watsopanowu wachita zochulukirapo (kutha kulemba zomveka zomveka bwino ku Lua mothandizidwa ndi cURL, SSH ndi SNMP) ndikukhazikitsa magwiridwe antchito omwe amakupatsani mwayi wowongolera mayendedwe a script (pangani, sinthani, kuwongolera mtundu). , kufufuta ndi kusunga).

Interface yogwira ntchito ndi autohealing scripts. Seva yambitsaninso script kudzera pa SSH.

Zotsatira Zofunikira

Panthawi yoyendetsa, nkhani za ogwiritsa ntchito zidapangidwanso zomwe zimathandizira magwiridwe antchito apano ndikuwonjezera mtengo kwa kasitomala, nazi zina mwazo:

• khazikitsani kuthekera kotumiza zosinthika molunjika kuchokera ku chenjezo kupita ku autohealing script;
• onjezani chilolezo papulatifomu kudzera pa Active Directory.

Ndipo tidalandira zovuta zapadziko lonse lapansi - "kumanga" malonda ndi kuthekera kwina:

• Kumanga mokhazikika kwachitsanzo chogwiritsa ntchito zothandizira kutengera ML, osati malamulo ndi othandizira (mwinamwake vuto lalikulu tsopano);
• kuthandizira pazowonjezera zolembera ndi zomveka (ndipo izi zidzakhala JavaScript).

Mu lingaliro langa, chofunika kwambiriZomwe woyendetsa uyu akuwonetsa ndi zinthu ziwiri:

1. Kugwirizana ndi kasitomala ndi chinsinsi chakuchita bwino, pamene kulankhulana kogwira mtima kumamangidwa pamaziko a kukhulupirika ndi kumasuka, ndipo kasitomala amakhala gawo la gulu lomwe limapeza zotsatira zazikulu mu nthawi yochepa.
2. Mulimonse momwe zingakhalire, ndikofunikira "kusintha" ndikumanga "ndodo" - njira zothetsera mavuto. Ndi bwino kuthera nthawi yochulukirapo, koma pangani njira yothetsera vutoli yomwe idzagwiritsidwe ntchito ndi makasitomala ena. Mwa njira, izi ndi zomwe zidachitika, pulogalamu yowonjezera ndikuchotsa kudalira kwa Azure kunapereka mtengo wowonjezera kwa makasitomala ena (hello, Federal Law 152).

Source: www.habr.com