Kuvandudza maindasitiri emasoftware masisitimu kunoda kutarisisa kukuru kune kukanganisa kushivirira kwechigadzirwa chekupedzisira, pamwe nekukurumidza kupindura kune kukundikana uye kutadza kana zvikaitika. Kuongorora, hongu, kunobatsira kupindura kukundikana uye kukundikana zvakanyanya uye nekukurumidza, asi hazvina kukwana. Kutanga, zvakanyanya kuoma kuchengeta nhamba huru yemaseva - nhamba huru yevanhu inodiwa. Chechipiri, iwe unofanirwa kuve nekunzwisisa kwakanaka kwemashandisirwo anoita application kuitira kufanotaura mamiriro ayo. Naizvozvo, isu tinoda vanhu vazhinji vane kunzwisisa kwakanaka kwemasisitimu atiri kugadzira, maitiro avo uye maitiro. Ngatitorei kuti kunyangwe ukawana vanhu vakakwana vanoda kuita izvi, zvinotora nguva yakawanda kuvadzidzisa.
Kuita sei? Apa ndipo panouya hungwaru hwekugadzira kuzotibatsira. Nyaya yacho ichataura nezvayo (predictive maintenance). Iyi nzira iri kushingaira kuwana mukurumbira. Nhamba huru yezvinyorwa zvakanyorwa, kusanganisira paHabrΓ©. Makambani makuru anoshandisa zvizere nzira iyi kuchengetedza kushanda kwemaseva avo. Pashure pokudzidza nhamba huru yezvinyorwa, takasarudza kuedza nzira iyi. Chii chakabuda?
Nhanganyaya
Iyo yakagadziridzwa software system nekukurumidza kana gare gare inotanga kushanda. Izvo zvakakosha kune mushandisi kuti sisitimu inoshanda pasina kukundikana. Kana emergency ikaitika, inofanirwa kugadziriswa nekunonoka kudiki.
Kurerutsa rutsigiro rwehunyanzvi hwesoftware system, kunyanya kana paine maseva mazhinji, zvirongwa zvekutarisa zvinowanzo shandiswa zvinotora metrics kubva kune inomhanya software system, inoita kuti zvikwanise kuongorora mamiriro ayo uye kubatsira kuona kuti chii chaizvo chakonzera kutadza. Maitiro aya anonzi software system monitoring.
Mufananidzo 1. Grafana monitoring interface
Metrics zviratidzo zvakasiyana-siyana zvesoftware system, nharaunda yekuuraya, kana komputa yepanyama pasi iyo sisitimu iri kushanda ine chidhindo chenguva iyo metric yakagamuchirwa. Mukuongorora kwakamira, aya metrics anonzi nguva akateedzana. Kuti utarise mamiriro esoftware system, metrics inoratidzwa muchimiro chegirafu: nguva iri paX axis, uye kukosha kuri padivi peY axis (Mufananidzo 1). Zviuru zvinoverengeka zvemetrics zvinogona kutorwa kubva kune inomhanya software system (kubva kune imwe neimwe node). Vanoumba nzvimbo yemametrics (multidimensional time series).
Sezvo huwandu hukuru hwemetrics hunounganidzwa kune yakaoma software masisitimu, kutarisisa kwemanyorero kunova basa rakaoma. Kudzikisa huwandu hwe data yakaongororwa nemutungamiriri, maturusi ekutarisa ane maturusi ekuona otomatiki matambudziko anogona kuitika. Semuenzaniso, unogona kugadzira chinokonzeresa moto kana yemahara dhisiki nzvimbo inowira pazasi pechikumbaridzo chakatarwa. Iwe unogona zvakare kuongorora otomatiki kuvharika kwesevha kana kuderera kwakasimba mukumhanya kwebasa. Mukuita, maturusi ekutarisa anoita basa rakanaka rekuona kutadza kwakatoitika kana kuziva zviratidzo zviri nyore zvekutadza kweramangwana, asi kazhinji, kufanotaura kukundikana kunogoneka kunoramba kwakaoma kutsemuka kwavari. Kufanotaura kuburikidza nekuongorora kwemaoko kwemetrics kunoda kubatanidzwa kwenyanzvi dzinokwanisa. Kubereka kwakaderera. Zvizhinji zvingangokundikana zvingasaonekwa.
Munguva pfupi yapfuura, iyo inonzi inofanotaura kuchengetedza kwesoftware masisitimu yave kuwedzera mukurumbira pakati pemakambani makuru ekuvandudza software yeIT. Izvo zvakakosha zveiyi nzira ndeyekutsvaga matambudziko anotungamira mukuparara kwehurongwa mumatanho ekutanga, isati yakundikana, uchishandisa njere dzekugadzira. Iyi nzira haibvisi zvachose kutariswa kwemanyorerwo ehurongwa. Inobatsira pakuongororwa kwese.
Chishandiso chikuru chekushandisa kufanotaura kuchengetedza ibasa rekutsvaga anomalies munhevedzano yenguva, kubvira pakaitika anomaly mune data pane mukana mukuru wekuti mushure menguva yakati pachava nekukundikana kana kukundikana. Anomaly imwe kutsauka mukuita kwesoftware system, senge kuona kudzikisirwa mukumhanyisa kwekuita kweimwe rudzi rwechikumbiro kana kudzikira kweavhareji yenhamba yezvikumbiro zvinoitirwa padanho rinogara riri remaklayiti.
Iro basa rekutsvaga anomalies yesoftware masisitimu ane zvawo chaiwo. Mune dzidziso, kune yega yega software system inofanirwa kugadzira kana kunatsiridza nzira dziripo, sezvo kutsvaga kweanomalies kunoenderana zvakanyanya nedata razvinoitwa, uye data yemasoftware system inosiyana zvakanyanya zvichienderana nemidziyo yekushandisa system. , kusvika kune komputa yairi kushanda pairi.
Nzira dzekutsvaga anomalies kana uchifanotaura kutadza kwemasoftware system
Chokutanga pane zvose, zvakakodzera kutaura kuti pfungwa yekufanotaura kukundikana yakafuridzirwa nechinyorwa . Kuedza kushanda kweiyo nzira nekutsvaga otomatiki kweanomalies, iyo Web-Consolidation software system yakasarudzwa, inova imwe yemapurojekiti ekambani yeNPO Krista. Kare, kutariswa kwemaoko kwaiitirwa iyo zvichienderana nemametric akagamuchirwa. Sezvo sisitimu yacho yakanyanya kuoma, nhamba huru yemametric inotorwa nokuda kwayo: JVM zviratidzo (mutakuri wemarara), zviratidzo zveOS iyo iyo code inotevedzwa (virtual memory, % OS CPU mutoro), network zviratidzo (network load. ), sevha pachayo (CPU mutoro, ndangariro), wildfly metrics uye mashandisirwo ega metrics kune ese akakosha subsystems.
Yese metrics inotorwa kubva kuhurongwa uchishandisa graphite. Pakutanga, dhatabhesi rezevezeve raishandiswa seyakajairwa mhinduro yegrafana, asi sezvo base yevatengi yakawedzera, graphite yakanga isisagone kurarama, yapedza simba reiyo DC disk subsystem. Mushure meizvi, zvakasarudzwa kutsvaga mhinduro inoshanda. Sarudzo yakaitwa mukufarira , iyo yakaita kuti zvibvirire kuderedza mutoro padhisiki subsystem nekuraira kwehukuru uye kuderedza yakagarwa dhisiki nzvimbo nekashanu kusvika katanhatu. Pazasi pane dhayagiramu yemaitiro ekuunganidza metrics uchishandisa graphite + clickhouse (Mufananidzo 2).
Mufananidzo 2. Scheme yekuunganidza metrics
Dhiagiramu inotorwa kubva mukati zvinyorwa. Inoratidza kutaurirana pakati pegrafana (iyo yekutarisa UI yatinoshandisa) uye graphite. Kubvisa metrics kubva pachikumbiro kunoitwa neyakasiyana software - . Anodziisa mugraphite.
Iyo Web Consolidation system ine akati wandei maficha anogadzira matambudziko ekufanotaura kukundikana:
- Chimiro chacho chinowanzochinja. Shanduro dzakasiyana siyana dziripo kune iyi software system. Imwe neimwe yadzo inounza shanduko kune software chikamu chehurongwa. Saizvozvo, nenzira iyi, vanogadzira vanokanganisa zvakananga metrics yeakapihwa system uye vanogona kukonzera shanduko yemaitiro;
- chiitiko chekushandisa, pamwe chete nezvinangwa izvo vatengi vanoshandisa iyi sisitimu, kazhinji inokonzeresa kusakanganiswa pasina kuderedzwa kwekare;
- iyo muzana yeanomalies maererano neseti yese data idiki (<5%);
- Panogona kunge paine mikaha mukugamuchira zviratidzo kubva kuhurongwa. Mune dzimwe nguva pfupi, iyo yekutarisa system inotadza kuwana metrics. Semuenzaniso, kana sevha yakawandisa. Izvi zvakakosha pakudzidzisa neural network. Pane kudiwa kuzadza mapeji synthetically;
- Mhosva dzine anomalies dzinowanzoshanda kune yakatarwa zuva/mwedzi/nguva (mwaka). Iyi sisitimu ine mitemo yakajeka yekushandiswa kwayo nevashandisi. Saizvozvo, ma metrics anoshanda chete kune yakatarwa nguva. Iyo sisitimu haigone kushandiswa nguva dzose, asi mune mimwe mwedzi chete: kusarudza zvichienderana negore. Mamiriro ezvinhu anomuka apo maitiro akafanana emametrics mune imwe nyaya anogona kutungamira mukukundikana kwesoftware system, asi kwete mune imwe.
Kutanga, nzira dzekuona kusawirirana mukutarisa data yemasoftware system dzakaongororwa. Muzvinyorwa zvenyaya iyi, kana chikamu cheanomalies chiri chidiki kune yese seti yedata, inowanzo kurudzirwa kushandisa neural network.
Iyo yakakosha pfungwa yekutsvaga anomalies uchishandisa neural network data inoratidzwa mumufananidzo 3:
Mufananidzo 3. Kutsvaga anomalies uchishandisa neural network
Zvichienderana nemhedzisiro yekufanotaura kana kudzoreredzwa kwehwindo rekuyerera kwemazuva ano kwema metrics, kutsauka kubva kune yakagamuchirwa kubva kune inomhanya software system inoverengerwa. Kana paine musiyano mukuru pakati pemetrics inowanikwa kubva kusoftware system uye neural network, tinogona kugumisa kuti chikamu chezvino data hachishamisi. Aya anotevera akateedzana matambudziko anomuka ekushandiswa kweneural network:
- kushanda nemazvo mukutepfenyura modhi, iyo data yekudzidzira neural network modhi inofanira kusanganisira chete "yakajairika" data;
- zvinodikanwa kuve nemhando yemazuva ano yekuonekwa kwayo. Kuchinja maitiro uye mwaka mune metrics kunogona kukonzera nhamba yakakura yenhema yakanaka mumuenzaniso. Kuti uigadzirise, zvinodikanwa kunyatsoona nguva iyo iyo modhi yakapera. Kana iwe ukavandudza muenzaniso gare gare kana kuti mberi, saka, zvichida, nhamba huru yenhema inozotevera.
Isu hatifanirewo kukanganwa nezve kutsvaga nekudzivirira kuwanzoitika kwenhema. Zvinotendwa kuti zvinowanzoitika mumamiriro ekukurumidzira. Nekudaro, ivo vanogona zvakare kuve mhedzisiro yekukanganisa neural network nekuda kwekusakwana kudzidziswa. Izvo zvinodiwa kuderedza nhamba yezvinyorwa zvenhema zvemuenzaniso. Zvikasadaro, kufembera kwenhema kunoparadza yakawanda yenguva yemutungamiriri yakagadzirirwa kutarisa sisitimu. Nenguva isipi maneja anongomira kupindura kune "paranoid" yekutarisa system.
Recurrent neural network
Kuti uone anomalies munguva dzakatevedzana, unogona kushandisa ine LSTM ndangariro. Dambudziko chete nderekuti rinogona kushandiswa chete kune yakafanotaurwa nguva yakatevedzana. Muchiitiko chedu, haasi ese metrics anofanotaurwa. Kuedza kushandisa RNN LSTM kune imwe nguva inoratidzwa mumufananidzo 4.
Mufananidzo 4. Muenzaniso wekudzokororwa neural network neLSTM memory cells
Sezvinoonekwa kubva Mufananidzo 4, RNN LSTM yakakwanisa kubata nekutsvaga kweanomalies munguva ino. Iko mhedzisiro ine yakakwira yekufembera kukanganisa (kureva kukanganisa), anomaly muzviratidziro zvaitika. Kushandisa imwe chete RNN LSTM kuchave pachena kuti haina kukwana, nekuti inoshanda kune mashoma mametrics. Inogona kushandiswa senzira yekubatsira yekutsvaga anomalies.
Autoencoder yekufembera kukundikana
- chaizvo artificial neural network. Iyo yekupinza layer ndeye encoder, inobuda layer idecoder. Izvo zvakashata zveese neural network zverudzi urwu ndezvekuti ivo havaite zvemuno anomalies zvakanaka. A synchronous autoencoder architecture yakasarudzwa.
Mufananidzo 5. Muenzaniso wekushanda kwe autoencoder
Autoencoders inodzidziswa pane yakajairwa data uye wobva wawana chimwe chinhu chisinganzwisisike mune data inopihwa kune modhi. Ndizvo zvauri kuda pabasa iri. Chasara kusarudza kuti ndeipi autoencoder inokodzera basa iri. Iyo yekuvaka yakapusa fomu ye autoencoder ndeye kumberi, isingadzoke neural network, yakafanana zvakanyanya ne (multilayer perceptron, MLP), ine yekupinza layer, yekubuda layer, uye imwe kana anopfuura akavanzika akaturikidzana anovabatanidza.
Nekudaro, mutsauko uripo pakati peautoencoder neMLPs ndewekuti mune autoencoder, iyo inobuda layer ine nhamba imwechete yemanodhi seyekuisa layer, uye kuti pachinzvimbo chekudzidziswa kufanotaura kukosha kwechinangwa Y chakapihwa nekuisa X, autoencoder inodzidziswa. kugadzira patsva maXs ayo. Naizvozvo, Autoencoders mamodheru ekudzidza asina kutariswa.
Basa re autoencoder ndere kutsvaga ma indices enguva r0 ... rn inoenderana neasinganzwisisike zvinhu zviri muinput vector X. Izvi zvinogoneka nekutsvaga kukanganisa kwakapetwa.
Mufananidzo 6. Synchronous autoencoder
Nokuti iyo autoencoder yakasarudzwa . Zvakanakira: kugona kushandisa kutenderera kugadzirisa modhi uye ishoma nhamba yeneural network paramita kana ichienzaniswa nemamwe mavakirwo.
Mechanism yekudzikisa nhema dzenhema
Nekuda kwekuti mamiriro akasiyana-siyana asina kujairika anomuka, pamwe chete nemamiriro angangoita ekusakwana kudzidziswa kweneural network, yeanomaly yekuona modhi iri kugadzirwa, zvakasarudzwa kuti zvaive zvakafanira kugadzira nzira yekudzikisa manyepo. Iyi meshini yakavakirwa pachigadziko chetemplate icho chinorongedzerwa nemutungamiriri.
(DTW algorithm, kubva kuChirungu dynamic time warping) inokutendera iwe kuti uwane iyo yakakwana tsamba pakati pekutevedzana kwenguva. Kutanga kushandiswa mukuzivikanwa kwekutaura: inoshandiswa kuona kuti zviratidzo zviviri zvekutaura zvinomiririra sei mutsara umwechete unotaurwa. Mushure meizvozvo, chikumbiro chakawanikwa kune dzimwe nzvimbo.
Nheyo huru yekudzikisira zvibodzwa zvenhema kuunganidza dhatabhesi yezviyero nerubatsiro rwemushandisi anoronga nyaya dzinofungirwa dzinoonekwa pachishandiswa neural network. Tevere, chiyero chakasarudzika chinofananidzwa nechiitiko chakaonekwa nehurongwa, uye mhedziso inoitwa yekuti nyaya yacho ndeyenhema here kana kuti inotungamira mukukundikana. Iyo DTW algorithm inoshandiswa chaizvo kuenzanisa maviri nguva akateedzana. Iyo huru yekudzikisa chishandiso ichiri kupatsanura. Zvinotarisirwa kuti mushure mekuunganidza nhamba huru yezvinyorwa zvekutsvaga, hurongwa huchatanga kubvunza mushandisi zvishoma nekuda kwekufanana kwezviitiko zvakawanda uye kuitika kwezvimwe zvakafanana.
Nekuda kweizvozvo, zvichibva pane neural network nzira dzakatsanangurwa pamusoro, chirongwa chekuyedza chakavakwa kufanotaura kukundikana kweiyo "Web-Consolidation" system. Chinangwa chechirongwa ichi chaive, kushandisa dura raivepo rekutarisisa data uye ruzivo nezve kutadza kwapfuura, kuongorora kugona kweiyi nzira yemasoftware edu masisitimu. Chirongwa chechirongwa chinoratidzwa pazasi mufananidzo 7.
Mufananidzo 7. Kukundikana kufanotaura chirongwa chinobva pane metric space analysis
Mudhayagiramu, mabhuraki maviri makuru anogona kusiyaniswa: kutsvaga kwenguva isinganzwisisike muyero yekutarisisa data (metrics) uye nzira yekudzikisa manyepo enhema. Ongorora: Nekuda kwekuedza, iyo data inowanikwa kuburikidza neJDBC yekubatanidza kubva kune dhatabhesi umo graphite ichaichengeta.
Izvi zvinotevera kuwirirana kwegadziriro yekuongorora yakawanikwa semugumisiro wekusimudzira (Mufananidzo 8).
Mufananidzo 8. Interface yekuedza kuongorora system
Iyo interface inoratidza iyo muzana yeanomaly zvichienderana neakagamuchirwa metrics. Muchiitiko chedu, risiti inofananidzwa. Isu tatova nedata rese kwemavhiki akati wandei uye tiri kurodha zvishoma nezvishoma kuti titarise nyaya yeanomaly inotungamira mukukundikana. Iyo yepasi mamiriro bar inoratidza iyo yese muzana yedata inomaly panguva yakatarwa, iyo inotemerwa uchishandisa autoencoder. Zvakare, chikamu chakasiyana chinoratidzwa kune zvakafanotaurwa metrics, iyo inoverengerwa neRNN LSTM.
Muenzaniso wekuonekwa kusinganzwisisike kwakavakirwa pakuita kweCPU uchishandisa iyo RNN LSTM neural network (Mufananidzo 9).
Mufananidzo 9. RNN LSTM kuwanikwa
Mhosva yakapusa, yakajairika, asi ichitungamira kukutadza kwehurongwa, yakaverengerwa zvakabudirira uchishandisa RNN LSTM. Chiratidzo cheanomaly munguva ino yenguva i85-95%; zvese zviri pamusoro pe80% (chikumbaridzo chakatemerwa kuyedza) chinotorwa seanomaly.
Muenzaniso wekuonekwa kweanomaly apo sisitimu yatadza kutanga mushure mekuvandudzwa. Iyi mamiriro anoonekwa ne autoencoder (Mufananidzo 10).
Mufananidzo 10. Muenzaniso wekutsvaga autoencoder
Sezvauri kuona kubva pamufananidzo, PermGen yakanamatira pane imwe nhanho. Iyo autoencoder yakaona izvi zvinoshamisa nekuti yakanga isati yamboona zvakadaro. Pano iyo anomaly inoramba iri 100% kusvika iyo system yadzokera kune inoshanda. A anomaly anoratidzwa kune ese metrics. Sezvambotaurwa, iyo autoencoder haigone kuisa zvinokanganisa. Mushandi anoshevedzwa kuti aite basa iri mumamiriro ezvinhu aya.
mhedziso
PC "Web-Consolidation" yave ichigadzirwa kwemakore akati wandei. Iyo sisitimu iri munzvimbo yakagadzikana, uye nhamba yezviitiko zvakanyorwa idiki. Nekudaro, zvaive zvichigoneka kuwana anomalies inotungamira mukukundikana 5 - 10 maminetsi kutadza kusati kwaitika. Mune zvimwe zviitiko, kuzivisa kukundikana pachine nguva kunobetsera kuchengetedza nguva yakarongwa yakagoverwa kuita βbasa rokugadzirisaβ.
Kubva pane zviedzo zvakaitwa, zvave kukasika kuti utore mhedziso dzekupedzisira. Kusvika pari zvino, mhedzisiro yacho inopesana. Kune rimwe divi, zviri pachena kuti algorithms yakavakirwa neural network inokwanisa kuwana "inobatsira" anomalies. Nekune rumwe rutivi, pachine chikamu chikuru chezviyero zvenhema, uye hazvisi zvese zvinokanganisa zvakaonekwa nenyanzvi inokwanisa mune neural network inogona kuonekwa. Izvo zvisingabatsiri zvinosanganisira chokwadi chekuti ikozvino neural network inoda kudzidziswa nemudzidzisi kune yakajairika kushanda.
Kuti uwedzere kukudziridza iyo yekutadza kufanotaura sisitimu uye kuiunza kune inogutsa mamiriro, nzira dzinoverengeka dzinogona kufungidzirwa. Uku ndiko kuongororwa kwakadzama kwemakesi ane anomalies anotungamira mukukundikana, nekuda kwekuwedzera uku kune runyorwa rweakakosha metrics anopesvedzera zvakanyanya mamiriro ehurongwa, uye kuraswa kweasina kufanira ayo asingaikonzerese. Zvakare, kana tikafamba munzira iyi, tinogona kuedza kuyedza maalgorithms zvakanangana nematare edu ane anomalies anotungamira mukukundikana. Pane imwe nzira. Uku ndiko kuvandudzwa kweneural network architectures uye nekudaro kuwedzera kurongeka kwekuona nekudzikiswa kwenguva yekudzidziswa.
Ndinotaura kuonga kwangu kune vandinoshanda navo vakandibatsira kunyora nekuchengetedza kukosha kwechinyorwa ichi: uye Sergei Finogenov.
Source: www.habr.com