Kukula kwa mafakitale a mapulogalamu a mapulogalamu kumafuna chidwi chachikulu pa kulekerera zolakwika za mankhwala omaliza, komanso kuyankha mofulumira ku zolephera ndi zolephera ngati zikuchitika. Kuwunika, ndithudi, kumathandiza kuyankha zolephera ndi zolephera bwino komanso mofulumira, koma osakwanira. Choyamba, ndizovuta kwambiri kusunga ma seva ambiri - chiwerengero chachikulu cha anthu chikufunika. Kachiwiri, muyenera kumvetsetsa bwino momwe pulogalamuyo imagwirira ntchito kuti mulosere momwe ilili. Chifukwa chake, timafunikira anthu ambiri omwe amamvetsetsa bwino machitidwe omwe tikupanga, machitidwe awo ndi mawonekedwe awo. Tiyerekeze kuti ngakhale mutapeza anthu okwanira kuchita zimenezi, zimatengerabe nthawi yochuluka kuwaphunzitsa.
Zoyenera kuchita? Apa ndipamene nzeru zopangapanga zimatithandizira. Nkhaniyi ifotokoza (kukonza zolosera). Njirayi ikuyamba kutchuka. Zolemba zambiri zalembedwa, kuphatikiza pa HabrΓ©. Makampani akuluakulu amagwiritsa ntchito njira imeneyi kuti apitirize kugwira ntchito za maseva awo. Titaphunzira nkhani zambiri, tinaganiza zoyesa njira imeneyi. Chinabwera ndi chiyani?
Mau oyamba
Mapulogalamu opangidwa posakhalitsa amayamba kugwira ntchito. Ndikofunikira kwa wosuta kuti dongosolo lizigwira ntchito popanda zolephera. Ngati vuto lachitika mwadzidzidzi, liyenera kuthetsedwa mwachangu.
Kuti muchepetse chithandizo chaukadaulo cha pulogalamu yamapulogalamu, makamaka ngati pali ma seva ambiri, mapulogalamu oyang'anira nthawi zambiri amagwiritsidwa ntchito omwe amatenga ma metric kuchokera pamapulogalamu oyendetsa, kuti athe kudziwa momwe alili ndikuthandizira kudziwa chomwe chinayambitsa kulephera. Njirayi imatchedwa kuwunika kwamapulogalamu.
Chithunzi 1. Grafana monitoring interface
Metrics ndizizindikiro zosiyanasiyana zamapulogalamu, malo omwe amagwirira ntchito, kapena kompyuta yakuthupi yomwe dongosololi likuyenda ndi chizindikiro cha nthawi yomwe ma metric adalandiridwa. Pakuwunika kokhazikika, ma metrics awa amatchedwa mndandanda wanthawi. Kuwunika momwe pulogalamuyo ikuyendera, ma metric amawonetsedwa ngati ma graph: nthawi ili pa X axis, ndipo ma values ββali pamzere wa Y (Chithunzi 1). Ma metrics masauzande angapo amatha kutengedwa kuchokera pamapulogalamu oyendetsa (kuchokera ku node iliyonse). Amapanga danga la ma metrics (multidimensional time series).
Popeza kuti ma metric ambiri amasonkhanitsidwa pamapulogalamu ovuta, kuyang'anira pamanja kumakhala ntchito yovuta. Kuti muchepetse kuchuluka kwa deta yomwe yafufuzidwa ndi woyang'anira, zida zowunikira zimakhala ndi zida zodziwira zokha zovuta zomwe zingachitike. Mwachitsanzo, mutha kukonza choyambitsa moto pomwe malo aulere a disk agwera pansi pamlingo womwe watchulidwa. Mutha kudziwanso kuti kutsekedwa kwa seva kapena kutsika kwakukulu kwa liwiro la ntchito. MwachizoloΕ΅ezi, zida zowunikira zimagwira ntchito yabwino yozindikira zolephera zomwe zachitika kale kapena kuzindikira zizindikiro zosavuta za zolephera zamtsogolo, koma kawirikawiri, kuneneratu kulephera kotheka kumakhalabe mtedza wovuta kwa iwo. Kulosera kudzera mu kusanthula pamanja kwa ma metrics kumafuna kutengapo gawo kwa akatswiri oyenerera. Ndi zokolola zochepa. Zolephera zambiri zomwe zingatheke sizingadziwike.
Posachedwapa, zomwe zimatchedwa kukonzekera kwadongosolo kwa mapulogalamu a mapulogalamu zakhala zikudziwika kwambiri pakati pa makampani akuluakulu opanga mapulogalamu a IT. Chofunika kwambiri cha njirayi ndikupeza mavuto omwe amatsogolera kuwonongeka kwa dongosolo kumayambiriro, asanalephereke, pogwiritsa ntchito luntha lochita kupanga. Njirayi siyikuchotseratu kuyang'anira pamanja kwadongosolo. Ndiwothandizira pa ntchito yowunika zonse.
Chida chachikulu chothandizira kukonza zolosera ndi ntchito yofufuza zolakwika mumndandanda wanthawi, kuyambira pamene anomaly kumachitika mu deta pali mwayi waukulu kuti patapita nthawi padzakhala kulephera kapena kulephera. Kusokoneza ndi kupatuka kwina kwa machitidwe a pulogalamu yamapulogalamu, monga kuzindikira kuwonongeka kwa liwiro la mtundu umodzi wa pempho kapena kuchepa kwa kuchuluka kwa zopempha zomwe zimaperekedwa pafupipafupi pagawo lamakasitomala.
Ntchito yofufuza zolakwika pamapulogalamu apulogalamu ili ndi zake. Mwachidziwitso, pa pulogalamu iliyonse ya pulogalamuyo m'pofunika kupanga kapena kukonza njira zomwe zilipo, chifukwa kufufuza zolakwika kumadalira kwambiri deta yomwe ikuchitika, ndipo deta ya machitidwe a mapulogalamu amasiyana kwambiri malinga ndi zida zogwiritsira ntchito dongosolo. , mpaka pa kompyuta yomwe ikuyenda.
Njira zofufuzira zolakwika pakulosera kulephera kwa mapulogalamu apulogalamu
Choyamba, m'pofunika kunena kuti lingaliro la kulosera zolephera linauziridwa ndi nkhaniyi . Kuti muyese mphamvu ya njirayo pofufuza zosokoneza, pulogalamu ya Web-Consolidation inasankhidwa, yomwe ndi imodzi mwa ntchito za kampani ya NPO Krista. M'mbuyomu, kuyang'anira pamanja kunkachitika potengera ma metric omwe adalandira. Popeza dongosololi ndi lovuta kwambiri, ma metric ambiri amatengedwa: zizindikiro za JVM (zonyamula zinyalala), zizindikiro za OS zomwe codeyo imachitidwa (kukumbukira kwenikweni, % OS CPU katundu), zizindikiro za netiweki (network load). ), seva yokha (CPU load , memory), ma metrics owuluka zakuthengo ndi ma metric akugwiritsa ntchito pazinthu zonse zovuta.
Ma metric onse amatengedwa kuchokera kudongosolo pogwiritsa ntchito graphite. Poyambirira, nkhokwe ya manong'onoting'ono idagwiritsidwa ntchito ngati njira yothetsera grafana, koma pamene makasitomala amakula, graphite inalephera kupirira, popeza inatopetsa mphamvu ya disk subsystem ya DC. Zitatha izi, adaganiza zopeza njira yothandiza kwambiri. Chisankho chinapangidwa mokomera , zomwe zinapangitsa kuti kuchepetsa katundu pa disk subsystem ndi dongosolo la kukula ndi kuchepetsa danga la disk lokhalamo kasanu mpaka kasanu ndi kamodzi. Pansipa pali chithunzi cha makina osonkhanitsira ma metrics pogwiritsa ntchito graphite +clickhouse (Chithunzi 2).
Chithunzi 2. Dongosolo lotolera ma metric
Chithunzicho chimatengedwa kuchokera ku zolemba zamkati. Ikuwonetsa kulumikizana pakati pa grafana (uI yowunikira yomwe timagwiritsa ntchito) ndi graphite. Kuchotsa ma metric ku pulogalamu kumapangidwa ndi mapulogalamu osiyana - . Amawaika mu graphite.
Web Consolidation System ili ndi zinthu zingapo zomwe zimabweretsa mavuto pakulosera zolephera:
- Zomwe zimachitika nthawi zambiri zimasintha. Mabaibulo osiyanasiyana zilipo kwa dongosolo mapulogalamu. Aliyense wa iwo amabweretsa kusintha kwa pulogalamu ya pulogalamuyo. Chifukwa chake, mwanjira iyi, opanga amakhudza mwachindunji ma metrics a dongosolo lomwe laperekedwa ndipo angayambitse kusintha;
- mawonekedwe ogwiritsira ntchito, komanso zolinga zomwe makasitomala amagwiritsira ntchito dongosololi, nthawi zambiri zimayambitsa zolakwika popanda kuwonongeka koyambirira;
- kuchuluka kwa zosokoneza zokhudzana ndi deta yonse ndizochepa (<5%);
- Pakhoza kukhala mipata pakulandira zizindikiro kuchokera ku dongosolo. M'kanthawi kochepa, makina owunikira amalephera kupeza ma metric. Mwachitsanzo, ngati seva yadzaza. Izi ndizofunikira pakuphunzitsa neural network. M'pofunika kudzaza mipata synthetically;
- Milandu yokhala ndi zolakwika nthawi zambiri imakhala yoyenera pa tsiku/mwezi/nthawi (nyengo). Dongosololi lili ndi malamulo omveka bwino ogwiritsidwa ntchito ndi ogwiritsa ntchito. Chifukwa chake, ma metrics ndi othandiza pakanthawi kochepa. Dongosolo silingagwiritsidwe ntchito nthawi zonse, koma m'miyezi ina yokha: mosankha kutengera chaka. Mikhalidwe imabwera pamene khalidwe lofanana la ma metrics muzochitika zina zingayambitse kulephera kwa pulogalamu ya mapulogalamu, koma osati mwa wina.
Poyambira, njira zodziwira zolakwika pakuwunika deta yamapulogalamu apulogalamu zidawunikidwa. M'nkhani za mutuwu, pamene kuchuluka kwa zolakwika kumakhala kochepa poyerekeza ndi deta yonse, nthawi zambiri amalangizidwa kuti agwiritse ntchito neural network.
Mfundo yofunikira pakufufuza zolakwika pogwiritsa ntchito neural network data ikuwonetsedwa pa Chithunzi 3:
Chithunzi 3. Kufufuza zolakwika pogwiritsa ntchito neural network
Kutengera zotsatira za kuneneratu kapena kubwezeretsedwa kwa zenera la mayendedwe amakono a ma metrics, kupatuka komwe kumalandiridwa kuchokera ku pulogalamu yoyendetsera pulogalamu kumawerengedwa. Ngati pali kusiyana kwakukulu pakati pa ma metric omwe atengedwa kuchokera ku pulogalamu yamapulogalamu ndi neural network, titha kunena kuti gawo lomwe lilipo pano ndi lodabwitsa. Mavuto otsatirawa amabwera pakugwiritsa ntchito ma neural network:
- kuti mugwire ntchito moyenera mumayendedwe akukhamukira, deta yophunzitsira mitundu ya neural network iyenera kukhala ndi data "yabwinobwino" yokha;
- m'pofunika kukhala ndi chitsanzo chamakono kuti azindikire bwino. Kusintha mayendedwe ndi nyengo mu ma metrics kungayambitse kuchuluka kwa zabwino zabodza muzachitsanzo. Kuti musinthe, m'pofunika kudziwa bwino nthawi yomwe chitsanzocho chinatha. Ngati mutasintha chitsanzo pambuyo pake kapena kale, ndiye, mwinamwake, chiwerengero chachikulu cha zolakwika chidzatsatira.
Sitiyeneranso kuiwala za kufunafuna ndi kupewa kupezeka pafupipafupi kwa zinthu zabodza. Zimaganiziridwa kuti nthawi zambiri zidzachitika mwadzidzidzi. Komabe, zitha kukhalanso zotsatira za cholakwika cha neural network chifukwa chosaphunzitsidwa mokwanira. Ndikofunikira kuchepetsa chiwerengero cha zolakwika zachitsanzo. Kupanda kutero, zolosera zabodza zidzawononga nthawi yochuluka yoyang'anira kuti ayang'ane dongosolo. Posakhalitsa woyang'anira adzangosiya kuyankha ku "paranoid" yowunikira.
Neural network yokhazikika
Kuti muwone zolakwika muzotsatira zanthawi, mutha kugwiritsa ntchito ndi LSTM memory. Vuto lokhalo ndikuti litha kugwiritsidwa ntchito pazolosera nthawi. Kwa ife, si ma metric onse omwe amalosera. Kuyesa kugwiritsa ntchito RNN LSTM pamndandanda wanthawi kukuwonetsedwa pazithunzi 4.
Chithunzi 4. Chitsanzo cha neural network yobwerezabwereza ndi maselo a kukumbukira LSTM
Monga momwe tikuonera pa Chithunzi 4, RNN LSTM inatha kuthana ndi kufunafuna zolakwika mu nthawi ino. Kumene zotsatira zake zimakhala ndi zolakwika zolosera kwambiri (zolakwika zomwe zimatanthawuza), zolakwika mu zizindikiro zachitikadi. Kugwiritsa ntchito RNN LSTM imodzi sikungakhale kokwanira, chifukwa imagwira ntchito pama metrics ochepa. Itha kugwiritsidwa ntchito ngati njira yothandizira pakufufuza zolakwika.
Autoencoder yolosera zolephera
- makamaka neural neural network. Gawo lolowera ndi encoder, gawo lotulutsa ndi decoder. Choyipa cha ma neural network onse amtunduwu ndikuti samayika zosokoneza bwino. Zomanga za synchronous autoencoder zidasankhidwa.
Chithunzi 5. Chitsanzo cha ntchito ya autoencoder
Ma Autoencoder amaphunzitsidwa pa data wamba kenako amapeza china chake chodabwitsa mu data yomwe yaperekedwa ku mtunduwo. Zomwe mukufunikira pa ntchitoyi. Chotsalira ndikusankha autoencoder yomwe ili yoyenera pa ntchitoyi. Mapangidwe osavuta kwambiri a autoencoder ndi netiweki yamtsogolo, yosabwerera, yomwe ili yofanana kwambiri (multilayer perceptron, MLP), yokhala ndi gawo lolowera, gawo lotulutsa, ndi gawo limodzi kapena zingapo zobisika zowalumikiza.
Komabe, kusiyana pakati pa ma autoencoder ndi MLPs ndikuti mu autoencoder, gawo lotulutsa limakhala ndi nambala yofanana ndi gawo lolowera, komanso kuti m'malo mophunzitsidwa kulosera zandalama Y yoperekedwa ndi X, autoencoder imaphunzitsidwa. kuti amangenso ma X ake. Chifukwa chake, ma Autoencoders ndi mitundu yophunzirira yosayang'aniridwa.
Ntchito ya autoencoder ndikupeza zizindikiro za nthawi r0 ... rn zogwirizana ndi zinthu zosasangalatsa mu vector yolowetsa X. Izi zimatheka pofufuza zolakwika za squared.
Chithunzi 6. Synchronous autoencoder
Pakuti autoencoder idasankhidwa . Ubwino wake: kuthekera kogwiritsa ntchito njira yosinthira kukhamukira komanso kachulukidwe kakang'ono ka neural network magawo poyerekeza ndi zomangamanga zina.
Njira yochepetsera zabwino zabodza
Chifukwa chakuti zochitika zosiyanasiyana zachilendo zimachitika, komanso kuthekera kwa maphunziro osakwanira a neural network, chifukwa cha njira yodziwikiratu yomwe ikupangidwa, adaganiza kuti kunali koyenera kupanga njira yochepetsera zolakwika zabodza. Makinawa amachokera ku template yomwe imayikidwa ndi woyang'anira.
(DTW aligorivimu, kuchokera ku English dynamic time warping) imakupatsani mwayi wopeza makalata oyenera pakati pa kutsatizana kwa nthawi. Koyamba kugwiritsidwa ntchito pozindikira mawu: amagwiritsidwa ntchito kudziwa momwe zizindikiro ziwiri zoyankhulira zimayimira mawu omwewo omwe amalankhulidwa. Pambuyo pake, ntchitoyo idapezeka m'malo ena.
Mfundo yayikulu yochepetsera zabwino zabodza ndikusonkhanitsa nkhokwe ya miyezo mothandizidwa ndi wogwiritsa ntchito yemwe amagawa milandu yokayikitsa yopezeka pogwiritsa ntchito neural network. Kenako, mulingo wamaguluwo umayerekezedwa ndi zomwe zidapezeka, ndipo pamapeto pake amapangidwa ngati mlanduwo ndi wabodza kapena walephera. Algorithm ya DTW imagwiritsidwa ntchito ndendende kufanizira magawo awiri anthawi. Chida chachikulu chochepetsera chikadali chamagulu. Zikuyembekezeka kuti mutatha kusonkhanitsa milandu yambiri yofotokozera, dongosololi lidzayamba kufunsa wogwiritsa ntchito pang'ono chifukwa cha kufanana kwa zochitika zambiri komanso zochitika zofanana.
Chotsatira chake, pogwiritsa ntchito njira za neural network zomwe zafotokozedwa pamwambapa, pulogalamu yoyesera inamangidwa kuti iwonetsere zolephera za "Web-Consolidation" dongosolo. Cholinga cha pulojekitiyi chinali kugwiritsa ntchito nkhokwe yomwe ilipo yowunikira deta komanso zambiri za zolephera zam'mbuyomu, kuyesa luso la njirayi pamapulogalamu athu. Chiwembu cha pulogalamuyi chikuwonetsedwa pansipa mu Chithunzi 7.
Chithunzi 7. Chiwembu cholosera cholephera chotengera kusanthula kwa metric space
Pachithunzichi, midadada ikuluikulu ikuluikulu ingasiyanitsidwe: kusaka kwakanthawi kodabwitsa mumayendedwe owunikira (ma metrics) ndi njira yochepetsera zabwino zabodza. Zindikirani: Zolinga zoyesera, deta imapezedwa kudzera pa intaneti ya JDBC kuchokera ku database yomwe graphite idzasungiramo.
Zotsatirazi ndi mawonekedwe a njira yowunikira yomwe imapezeka chifukwa cha chitukuko (Chithunzi 8).
Chithunzi 8. Mawonekedwe a kachitidwe kowunika koyeserera
Chiwonetserochi chikuwonetsa kuchuluka kwa zolakwika kutengera ma metric omwe adalandira. M'malo mwathu, chiphasocho chimafananizidwa. Tili ndi data yonse kwa milungu ingapo ndipo tikuyikweza pang'onopang'ono kuti tiwone ngati pali vuto lomwe limabweretsa kulephera. Malo otsikirapo amawonetsa kuchuluka kwa data pa nthawi yake, yomwe imatsimikiziridwa pogwiritsa ntchito autoencoder. Komanso, peresenti yosiyana imawonetsedwa pazitsulo zonenedweratu, zomwe zimawerengedwa ndi RNN LSTM.
Chitsanzo cha kuzindikirika kosasinthika kutengera magwiridwe antchito a CPU pogwiritsa ntchito RNN LSTM neural network (Chithunzi 9).
Chithunzi 9. Kutulukira kwa RNN LSTM
Mlandu wosavuta, makamaka wamba wamba, koma wotsogolera kulephera kwadongosolo, adawerengedwa bwino pogwiritsa ntchito RNN LSTM. Chizindikiro chanthawi yayitali ndi 85-95%; chilichonse chomwe chili pamwamba pa 80% (chiwopsezo chidatsimikiziridwa moyesera) chimawonedwa ngati chosokoneza.
Chitsanzo cha kuzindikirika kwachilendo pamene dongosolo silinathe kuyambiranso pambuyo posintha. Izi zimadziwika ndi autoencoder (Chithunzi 10).
Chithunzi 10. Chitsanzo cha kuzindikira kwa autoencoder
Monga mukuwonera pachithunzichi, PermGen yakhazikika pamlingo umodzi. Autoencoder idapeza izi zachilendo chifukwa inali isanawonepo chilichonse chonga ichi. Apa anomaly amakhalabe 100% mpaka dongosolo libwerere ku ntchito. Chodabwitsa chikuwonetsedwa pama metrics onse. Monga tanena kale, autoencoder siyingatchule zolakwika. Wogwiritsa ntchito amafunsidwa kuti achite izi muzochitika izi.
Pomaliza
PC "Web-Consolidation" yakhala ikukula kwa zaka zingapo. Dongosololi lili mumkhalidwe wokhazikika, ndipo kuchuluka kwa zochitika zolembedwa ndizochepa. Komabe, zinali zotheka kupeza zolakwika zomwe zimatsogolera kulephera kwa mphindi 5 - 10 kulephera kusanachitike. NthaΕ΅i zina, kudziΕ΅itsa za kulephera kungathandize kusunga nthaΕ΅i yoikidwiratu yogwira ntchito βyokonzaβ.
Kutengera zoyeserera zomwe zidachitika, ndi molawirira kwambiri kuti tipeze mfundo zomaliza. Mpaka pano, zotsatira zake zikutsutsana. Kumbali imodzi, zikuwonekeratu kuti ma aligorivimu otengera ma neural network amatha kupeza zovuta "zothandiza". Kumbali inayi, pamakhalabe kuchuluka kwazabodza, ndipo sizovuta zonse zomwe akatswiri odziwa bwino pa neural network angadziwike. Zoyipa zake zikuphatikizapo mfundo yakuti tsopano neural network imafuna kuphunzitsidwa ndi mphunzitsi kuti azigwira ntchito bwino.
Kuti mupitilize kukulitsa dongosolo lolosera zolephera ndikupangitsa kuti likhale labwino, njira zingapo zitha kuganiziridwa. Uku ndikuwunikanso mwatsatanetsatane milandu yomwe ili ndi zolakwika zomwe zimapangitsa kulephera, chifukwa chowonjezera pamndandanda wazinthu zofunika kwambiri zomwe zimakhudza kwambiri dongosolo la dongosolo, ndikutaya zosafunikira zomwe sizikhudza. Komanso, ngati tisunthira mbali iyi, titha kuyesa kuyika ma aligorivimu makamaka pamilandu yathu yomwe ili ndi zolakwika zomwe zimadzetsa kulephera. Palinso njira ina. Uku ndikuwongolera kamangidwe ka neural network ndipo potero kumawonjezera kulondola kwa kuzindikira ndikuchepetsa nthawi yophunzitsira.
Ndikuthokoza anzanga amene anandithandiza kulemba ndi kusunga kufunika kwa nkhaniyi: ndi Sergei Finogenov.
Source: www.habr.com