Masikhumbule ukuthi i-Elastic Stack isekelwe ku-database ye-Elasticsearch engahlobene, isixhumi esibonakalayo sewebhu ye-Kibana kanye nabaqoqi bedatha nabaprosesa (i-Logstash edume kakhulu, i-Beats ehlukahlukene, i-APM nezinye). Enye yezengezo ezinhle kuso sonke isitaki somkhiqizo esohlwini ukuhlaziya idatha kusetshenziswa ama-algorithms okufunda komshini. Esihlokweni siyaqonda ukuthi ayini lawa ma-algorithms. Ngicela ngaphansi kwekati.
Ukufunda ngomshini kuyisici esikhokhelwayo se-shareware Elastic Stack futhi sifakiwe ku-X-Pack. Ukuze uqale ukuyisebenzisa, vele wenze kusebenze isilingo sezinsuku ezingama-30 ngemva kokusifaka. Ngemuva kokuthi isikhathi sesilingo siphelelwe yisikhathi, ungacela usekelo ukuze usinwebe noma uthenge okubhalisile. Izindleko zokubhalisa azibalwa ngokusekelwe kumthamo wedatha, kodwa enanini lamanodi asetshenzisiwe. Cha, umthamo wedatha, yiqiniso, uthinta inani lama-node adingekayo, kodwa noma kunjalo le ndlela yokulayisensa inomusa kakhulu ngokuphathelene nesabelomali senkampani. Uma singekho isidingo sokukhiqiza okuphezulu, ungonga imali.
I-ML ku-Elastic Stack ibhalwe ngo-C++ futhi isebenza ngaphandle kwe-JVM, lapho i-Elasticsearch ngokwayo isebenza khona. Okusho ukuthi, inqubo (ngendlela, ibizwa ngokuthi i-autodetect) idla yonke into i-JVM engagwinyi. Esimeni sedemo lokhu akubalulekile kangako, kodwa endaweni yokukhiqiza kubalulekile ukwaba amanodi ahlukene emisebenzi ye-ML.
Ama-algorithms okufunda komshini awela ezigabeni ezimbili β ΠΈ . Ku-Elastic Stack, i-algorithm isesigabeni "esingagadiwe". Ngu Ungabona izisetshenziswa zezibalo zama-algorithms okufunda komshini.
Ukuze kwenziwe ukuhlaziya, i-algorithm yokufunda yomshini isebenzisa idatha egcinwe kuzinkomba ze-Elasticsearch. Ungadala imisebenzi ukuze ihlaziywe kusuka kusixhumi esibonakalayo se-Kibana nange-API. Uma wenza lokhu nge-Kibana, awudingi ukwazi ezinye izinto. Isibonelo, izinkomba ezengeziwe ezisetshenziswa i-algorithm phakathi nokusebenza kwayo.
Izinkomba ezengeziwe ezisetshenziswa kunqubo yokuhlaziya.ml-state β ulwazi mayelana namamodeli ezibalo (izilungiselelo zokuhlaziya);
.ml-anomalies-* β imiphumela ye-ML algorithms;
.ml-notifications β izilungiselelo zezaziso ezisekelwe emiphumeleni yokuhlaziya.
Isakhiwo sedatha kusizindalwazi se-Elasticsearch siqukethe izinkomba namadokhumenti agcinwe kuzo. Uma iqhathaniswa nesizindalwazi esihlobene, inkomba ingafaniswa ne-schema yesizindalwazi, kanye nedokhumenti nerekhodi etafuleni. Lokhu kuqhathanisa kunemibandela futhi kunikezwa ukuze kube lula ukuqonda kokunye okubalulekile kulabo abake bezwa kuphela nge-Elasticsearch.
Ukusebenza okufanayo kuyatholakala nge-API njengokusebenzisa isixhumi esibonakalayo sewebhu, ngakho-ke ukuze kucace futhi kuqondwe imiqondo, sizobonisa indlela yokuyilungisa nge-Kibana. Kumenyu engakwesokunxele kunesigaba Sokufunda Ngomshini lapho ungakha khona Umsebenzi omusha. Ku-interface ye-Kibana kubukeka njengesithombe esingezansi. Manje sizohlaziya uhlobo ngalunye lomsebenzi futhi sibonise izinhlobo zokuhlaziya ezingakhiwa lapha.
Imethrikhi Eyodwa - ukuhlaziywa kwemethrikhi eyodwa, I-Multi Metric - ukuhlaziywa kwamamethrikhi amabili noma ngaphezulu. Kuzo zombili izimo, imethrikhi ngayinye ihlaziywa endaweni engayodwa, i.e. i-algorithm ayicabangi ukuziphatha kwamamethrikhi ahlaziywe ahambisanayo, njengoba kungase kubonakale endabeni ye-Multi Metric. Ukuze wenze izibalo ngokucabangela ukuhlobana kwamamethrikhi ahlukahlukene, ungasebenzisa ukuhlaziya inani labantu. Futhi Okuthuthukile kuhlela kahle ama-algorithms ngezinketho ezengeziwe zemisebenzi ethile.
I-Metric Eyodwa
Ukuhlaziya izinguquko kumethrikhi eyodwa yinto elula kakhulu engenziwa lapha. Ngemva kokuchofoza okuthi Dala Umsebenzi, i-algorithm izobheka okudidayo.
Ensimini I-Aggregation ungakhetha indlela yokusesha okudidayo. Ngokwesibonelo, nini min amanani angaphansi kwamanani ajwayelekile azobhekwa njengenqaba. Yidla Okukhulu, Okuphakeme Okushoyo, Okuphansi, Okuqondile, Okuhlukile nabanye. Izincazelo zayo yonke imisebenzi zingatholakala .
Ensimini Inkambu ikhombisa inkambu yezinombolo kudokhumenti esizokwenza ukuhlaziya kuyo.
Ensimini - imbudumbudu yezikhawu emugqeni wesikhathi lapho ukuhlaziya kuzokwenziwa khona. Ungathemba i-automation noma ukhethe mathupha. Isithombe esingezansi yisibonelo sombudumbudu ophansi kakhulu - ungase uphuthelwe indida. Usebenzisa lesi silungiselelo, ungashintsha ukuzwela kwe-algorithm kuya kokudidayo.
Ubude besikhathi bedatha eqoqiwe buyinto esemqoka ethinta ukusebenza kahle kokuhlaziya. Phakathi nokuhlaziywa, i-algorithm ikhomba izikhawu eziphindayo, ibala izikhawu zokuzethemba (imigqa eyisisekelo) futhi ihlonze okudidayo - ukuchezuka okungavamile kusukela ekuziphatheni okuvamile kwemethrikhi. Isibonelo nje:
Izisekelo ezinocezu oluncane lwedatha:
Uma i-algorithm inokuthile okumele ifunde kukho, isisekelo sibukeka kanjena:
Ngemva kokuqala umsebenzi, i-algorithm inquma ukuchezuka okuxakile kusuka kokujwayelekile futhi ikuklelise ngokwamathuba okuba ne-anomaly (umbala welebula elihambisanayo ukhonjiswe kubakaki):
Isexwayiso (okuluhlaza okwesibhakabhaka): ngaphansi kwama-25
Okuncane (okuphuzi): 25-50
Okukhulu (okuwolintshi): 50-75
Okubucayi (okubomvu): 75-100
Igrafu engezansi ibonisa isibonelo sokudidayo okutholiwe.
Lapha ungabona inombolo engu-94, ebonisa amathuba okuba ne-anomaly. Kuyacaca ukuthi njengoba inani lisondele ku-100, kusho ukuthi sine-anomaly. Ikholomu engezansi kwegrafu ibonisa amathuba amancane ngokudelela angu-0.000063634% yenani lemethrikhi elivela lapho.
Ngokungeziwe ekufuneni okudidayo, ungaqalisa ukubikezela e-Kibana. Lokhu kwenziwa kalula nangokubuka okufanayo nokudidayo - inkinobho isibikezelo ekhoneni eliphezulu kwesokudla.
Isibikezelo senziwa isikhathi esingamaviki angu-8 kusengaphambili. Ngisho noma ufuna ngempela, akusakwenzeki ngokuklama.
Kwezinye izimo, isibikezelo sizoba usizo kakhulu, isibonelo, lapho uqapha umthwalo wabasebenzisi engqalasizinda.
I-Multi Metric
Masiqhubekele esicini esilandelayo se-ML ku-Elastic Stack - sihlaziya amamethrikhi ambalwa kubheshi eyodwa. Kodwa lokhu akusho ukuthi ukuncika kwemethrikhi eyodwa kwenye kuzohlaziywa. Lokhu kuyafana ne-Single Metric, kodwa ngamamethrikhi amaningi esikrinini esisodwa ukuze kuqhathaniswe kalula umthelela wokukodwa kwesinye. Sizokhuluma ngokuhlaziya ukuncika kwemethrikhi eyodwa kwenye esigabeni Sesibalo sabantu.
Ngemva kokuchofoza isikwele esine-Multi Metric, kuzovela iwindi elinezilungiselelo. Ake sizibheke ngokuningiliziwe.
Okokuqala udinga ukukhetha izinkambu zokuhlaziya nokuhlanganiswa kwedatha kuzo. Izinketho zokuhlanganisa lapha ziyefana neze-Single Metric (Okukhulu, Okuphakeme Okushoyo, Okuphansi, Okuqondile, Okuhlukile nabanye). Ngaphezu kwalokho, uma uthanda, idatha ihlukaniswa ibe yinye yezinkambu (inkambu Hlukanisa Idatha). Esibonelweni, sikwenze lokhu ngenkambu I-OriginAirportID. Qaphela ukuthi igrafu yamamethrikhi kwesokudla manje yethulwa njengamagrafu amaningi.
Insimu Izinkambu ezingukhiye (Abathonya) kuthinta ngokuqondile okudidayo okutholiwe. Ngokuzenzakalela kuzohlala kukhona okungenani inani elilodwa lapha, futhi ungakwazi ukwengeza amanye. I-algorithm izocabangela umthelela walezi zinkambu lapho ihlaziya futhi ibonise amanani "anethonya" kakhulu.
Ngemuva kokwethulwa, into efana nale izovela kusixhumi esibonakalayo se-Kibana.
Lokhu okubizwa ngokuthi imephu yokushisa yokudidayo kwevelu ngayinye yenkambu I-OriginAirportID, esikubonise kuyo Hlukanisa Idatha. Njenge-Single Metric, umbala ubonisa izinga lokuchezuka okungavamile. Kulula ukwenza ukuhlaziya okufanayo, ngokwesibonelo, ezindaweni zokusebenza ukulandelela lezo ezinenani elikhulu ngokusolisayo lokugunyazwa, njll. Sesivele sabhala , okungabuye kuqoqwe futhi kuhlaziywe lapha.
Ngezansi kwemephu yokushisa kunohlu lokudidayo, kokunye ungashintshela ekubukweni kwe-Single Metric ukuze uthole ukuhlaziya okuningiliziwe.
Population
Ukuze ubheke okudidayo phakathi kokuhlobana phakathi kwamamethrikhi ahlukene, i-Elastic Stack inokuhlaziywa kwesibalo sabantu okukhethekile. Kungosizo lwayo ukuthi ungabheka amanani amangalisayo ekusebenzeni kweseva uma kuqhathaniswa nabanye lapho, ngokwesibonelo, inani lezicelo kuhlelo oluqondiwe likhuphuka.
Kulo mfanekiso, inkambu ethi Population ibonisa inani amamethrikhi ahlaziyiwe azohlotshaniswa nalo. Kulokhu yigama lenqubo. Njengomphumela, sizobona ukuthi umthwalo weprosesa wenqubo ngayinye ube nomthelela kanjani komunye nomunye.
Sicela uqaphele ukuthi igrafu yedatha ehlaziyiwe ihlukile ezimweni ezine-Single Metric kanye ne-Multi Metric. Lokhu kwenziwe e-Kibana ngokuklama kombono othuthukisiwe wokusatshalaliswa kwamanani edatha ehlaziyiwe.
Igrafu ibonisa ukuthi inqubo iziphathe ngendlela engavamile ukucindezeleka (ngendlela, ekhiqizwe insiza ekhethekile) kuseva iphuphu, othonye (noma okwavela ukuthi ungumgqugquzeli) ukwenzeka kwalokhu kudida.
Advanced
Izibalo ngokushuna kahle. Ngokuhlaziywa Okuthuthukile, izilungiselelo ezengeziwe zivela ku-Kibana. Ngemva kokuchofoza ithayela Okuthuthukile kumenyu yokudala, leli windi elinamathebhu liyavela. Ithebhu Imininingwane yomsebenzi Sikweqe ngenhloso, kunezilungiselelo eziyisisekelo ezingahlobene ngokuqondile nokusetha ukuhlaziya.
Π summary_count_field_name Ngokuzithandela, ungacacisa igama lenkambu kumadokhumenti aqukethe amanani ahlanganisiwe. Kulesi sibonelo, inani lezehlakalo ngomzuzu. IN ibonisa igama nenani lenkambu evela kudokhumenti equkethe inani elithile eliguquguqukayo. Usebenzisa imaski kulo mkhakha, ungahlukanisa idatha ehlaziyiwe ibe amasethi angaphansi. Naka inkinobho Engeza umtshina emfanekisweni odlule. Ngezansi umphumela wokuchofoza le nkinobho.
Nali ibhulokhi eyengeziwe yezilungiselelo zokumisa umtshina odidayo womsebenzi othile. Sihlela ukuxoxa ngezimo ezithile zokusetshenziswa (ikakhulukazi ezokuvikela) ezihlokweni ezilandelayo. Ngokwesibonelo, elinye lamacala ahlakaziwe. Ihlotshaniswa nosesho lwamanani angavamile ukuvela futhi isetshenziswa .
Ensimini umsebenzi Ungakhetha umsebenzi othize ukuze useshe okudidayo. Ngaphandle Okungajwayelekile, kunemisebenzi embalwa ethokozisayo - . Bahlonza okudidayo ekuziphatheni kwamamethrikhi usuku lonke noma iviki, ngokulandelana. Eminye imisebenzi yokuhlaziya .
Π inkambu_igama ikhombisa insimu yombhalo lapho ukuhlaziya kuzokwenziwa khona. Ngegama_lenkambu ingasetshenziswa ukuhlukanisa imiphumela yokuhlaziya yenani ngalinye lenkundla yedokhumenti ecaciswe lapha. Uma ugcwalisa igama_lenkambu uthola ukuhlaziywa kwenani labantu esixoxile ngakho ngenhla. Uma ucacisa inani ku partition_field_name, khona-ke kule nkambu yedokhumenti kuzobalwa izisekelo ezihlukene ngevelu ngayinye (inani lingaba, isibonelo, igama leseva noma inqubo kuseva). IN khipha_njalo angakhetha konke noma none, okuzosho ukungafaki (noma ukufaka) amanani enkambu yedokhumenti avela njalo.
Kulesi sihloko, sizame ukunikeza umbono omfushane ngangokunokwenzeka mayelana namakhono okufunda ngomshini ku-Elastic Stack; kusenemininingwane eminingi esele ngemuva kwezigcawu. Sitshele emazwaneni ukuthi yiziphi izimo okwazile ukuzixazulula usebenzisa i-Elastic Stack nokuthi uyisebenzisela miphi imisebenzi. Ukuze usithinte, ungasebenzisa imilayezo yomuntu siqu ku-HabrΓ© noma .
Source: www.habr.com