Ngatiyeukei kuti Elastic Stack yakavakirwa pane zvisiri zvehukama Elasticsearch dhatabhesi, iyo Kibana webhu interface uye vanounganidza data uye processors (yakakurumbira Logstash, akasiyana Beats, APM nevamwe). Imwe yeakanakisa yekuwedzera kune yese yakanyorwa chigadzirwa stack ndeyekuongorora data uchishandisa muchina kudzidza algorithms. Muchinyorwa tinonzwisisa kuti aya algorithms chii. Ndapota pasi pekati.
Kudzidza kwemuchina chinhu chakabhadharwa cheiyo shareware Elastic Stack uye inosanganisirwa muX-Pack. Kuti utange kuishandisa, ingo shandisa iyo 30-zuva kuyedzwa mushure mekuiswa. Mushure mekunge nguva yekuedza yapera, unogona kukumbira rutsigiro kuti uwedzere kana kutenga kunyoreswa. Mutengo wekunyorera unoverengerwa kwete zvichienderana nehuwandu hwe data, asi nenhamba yemanodhi anoshandiswa. Aiwa, huwandu hwe data, hongu, hunokanganisa huwandu hwemanodhi anodiwa, asi zvakadaro nzira iyi yekupa marezinesi inonyanya kuve nevanhu maererano nebhajeti rekambani. Kana pasina chikonzero chekugadzirwa kwepamusoro, unogona kuchengetedza mari.
ML muElastic Stack yakanyorwa muC ++ uye inomhanya kunze kweJVM, umo Elasticsearch pachayo inomhanya. Kureva, maitiro (nenzira, anonzi autodetect) anodya zvese izvo JVM isingamedze. Pademo inomira izvi hazvina kunyanya kunetsa, asi munzvimbo yekugadzira zvakakosha kugovera node dzakaparadzana dzeML mabasa.
Michina yekudzidza algorithms inowira mumapoka maviri - ΠΈ . MuElastic Stack, iyo algorithm iri muchikamu "chisina kutariswa". By Iwe unogona kuona iyo masvomhu zvishandiso zvemuchina kudzidza algorithms.
Kuita ongororo, muchina wekudzidza algorithm inoshandisa data yakachengetwa muElasticsearch indexes. Iwe unogona kugadzira mabasa ekuongorora zvese kubva kuKibana interface uye kuburikidza neAPI. Kana iwe ukaita izvi kuburikidza neKibana, saka haufanirwe kuziva zvimwe zvinhu. Semuenzaniso, mamwe indexes ayo algorithm inoshandisa panguva yekushanda kwayo.
Mamwe ma indices anoshandiswa mukuongorora maitiro.ml-state - ruzivo pamusoro pemhando dzenhamba (analysis settings);
.ml-anomalies-* - mhedzisiro yeML algorithms;
.ml-notifications - marongero ezviziviso zvinoenderana nemhedzisiro yekuongorora.
Iyo data chimiro muElasticsearch dhatabhesi ine indexes uye zvinyorwa zvakachengetwa mazviri. Kana ichienzaniswa nedatabase rehukama, index inogona kufananidzwa nedatabase schema, uye gwaro kune rekodhi mutafura. Kuenzanisa uku kune zvimiso uye kunopihwa kurerutsa kunzwisisa kwezvimwe zvinyorwa kune avo vakangonzwa nezve Elasticsearch.
Iko kushanda kwakafanana kunowanikwa kuburikidza neAPI sekuburikidza newebhu interface, saka kujekesa uye kunzwisisa kweiyo pfungwa, isu ticharatidza nzira yekuigadzirisa kuburikidza neKibana. Mumenu iri kuruboshwe kune chikamu cheKudzidza kweMichina apo iwe unogona kugadzira Basa idzva. MuKibana interface inoratidzika semufananidzo uri pasi apa. Iye zvino tichaongorora rudzi rwega rwebasa uye kuratidza marudzi ekuongorora anogona kuvakwa pano.
Single Metric - ongororo yemetric imwe, Multi Metric - kuongororwa kwemaviri kana anopfuura metrics. Muzviitiko zvese izvi, metric yega yega inoongororwa munzvimbo yakasarudzika, i.e. iyo algorithm haifungi nezve maitiro ezvakafanana akaongororwa metrics, sezvazvingaite senge mune yeMulti Metric. Kuita kuverenga uchifunga nezve kuwirirana kwemametric akasiyana, unogona kushandisa Population ongororo. Uye Yepamberi iri kunyatso-tuning maalgorithms ane dzimwe sarudzo dzemamwe mabasa.
Imwe Metric
Kuongorora shanduko mune imwechete metric ndicho chinhu chiri nyore chinogona kuitwa pano. Mushure mekudzvanya paGadzira Jobho, iyo algorithm ichatsvaga anomalies.
Mumunda Aggregation unogona kusarudza nzira yekutsvaga anomalies. Somuenzaniso, rini Min hunhu huri pazasi hwakajairwa hunhu hunozoonekwa sehusina kunaka. Eat Max, High Mean, Low, Mean, Distinct nevamwe. Tsananguro dzemabasa ese dzinogona kuwanikwa .
Mumunda munda inoratidza nhamba iri mugwaro ratichaita ongororo.
Mumunda - granularity yenguva dzenguva pane iyo nguva iyo ongororo ichaitwa. Iwe unogona kuvimba neotomatiki kana kusarudza nemaoko. Mufananidzo uri pazasi muenzaniso we granularity yakadzikira - unogona kupotsa iyo inomaly. Uchishandisa iyi gadziriro, unogona kushandura kunzwisiswa kwealgorithm kune anomalies.
Nguva ye data yakaunganidzwa chinhu chinokosha chinokanganisa kubudirira kwekuongorora. Panguva yekuongorora, algorithm inotaridza kudzokorora nguva, inoverengera nguva dzekuvimba (mabhesi) uye inotaridza zvisizvo - kutsauka kweatypical kubva kune yakajairika maitiro emetric. Somuenzaniso:
Baselines ine chidimbu chidiki che data:
Kana iyo algorithm iine chimwe chinhu chekudzidza kubva, iyo yekutanga inotaridzika seizvi:
Mushure mekutanga basa, iyo algorithm inosarudza kutsauka kusinganzwisisike kubva kune yakajairwa uye inoisa iwo zvinoenderana nemukana weanomaly (ruvara rwechinyorwa chinoenderana chinoratidzwa mumabharanzi):
Yambiro (bhuruu): isingasviki makumi maviri neshanu
Diki (yero): 25-50
Hukuru (orenji): 50-75
Yakakosha (tsvuku): 75-100
Girafu riri pazasi rinoratidza muenzaniso wekukanganisa kuwanikwa.
Pano unogona kuona nhamba 94, iyo inoratidza mukana weanomaly. Zviri pachena kuti sezvo kukosha kuri pedyo ne100, zvinoreva kuti tine anomaly. Iyo column pazasi pegirafu inoratidza iyo pejoratively diki mukana we 0.000063634% weiyo metric kukosha inoonekwa ipapo.
Pamusoro pekutsvaga anomalies, unogona kumhanya kufanotaura muKibana. Izvi zvinoitwa zviri nyore uye kubva kune imwechete maonero ane anomalies - bhatani kushurudzira mukona yekurudyi.
Kufanotaura kunoitwa kweanopfuura mavhiki masere pamberi. Kunyangwe iwe uchida chaizvo, hazvichagoneke nekugadzira.
Mune mamwe mamiriro ezvinhu, iyo fungidziro ichave yakakosha, semuenzaniso, kana uchitarisa mutoro wemushandisi pane zvivakwa.
Multi Metric
Ngatienderei kune inotevera ML chimiro muElastic Stack - tichiongorora akati wandei metrics mubatch rimwe. Asi izvi hazvireve kuti kuvimba kweimwe metric pane imwe kuchaongororwa. Izvi zvakafanana neSingle Metric, asi iine akawanda metrics pane imwe skrini kuitira nyore kuenzanisa kwemaitiro eimwe pane imwe. Tichataura nezve kuongorora kutsamira kweimwe metric pane imwe muchikamu chePopulation.
Mushure mekudzvanya pachikwere neMulti Metric, hwindo rine zvigadziriso richaonekwa. Ngatitarisei zvakadzama.
Kutanga iwe unofanirwa kusarudza minda yekuongorora uye data aggregation pairi. Sarudzo dzekubatanidza pano dzakafanana neyeSingle Metric (Max, High Mean, Low, Mean, Distinct nevamwe). Kupfuurirazve, kana zvichidikanwa, iyo data inokamurwa kuita imwe yeminda (munda Split Data) Mumuenzaniso, takaita izvi nemunda OriginAirportID. Ziva kuti metrics graph iri kurudyi yave kuratidzwa seakawanda magirafu.
munda Key Fields (Influences) inokanganisa zvakananga izvo zvakaonekwa anomalies. By default pachagara paine kamwechete kukosha pano, uye iwe unogona kuwedzera mamwe. Iyo algorithm ichafunga nezve pesvedzero yeminda iyi kana ichiongorora uye kuratidza yakanyanya "inopesvedzera" tsika.
Mushure mekutanga, chimwe chinhu chakadai chichaonekwa muKibana interface.
Izvi ndizvo zvinonzi kupisa mepu yeanomalies kune yega yega kukosha kwemunda OriginAirportID, zvatakaratidza Split Data. Sezvakaita Single Metric, ruvara runoratidza mwero wekutsauka. Zviri nyore kuita ongororo yakafanana, semuenzaniso, pane nzvimbo dzekushandira kuronda avo vane nhamba huru inofungidzira yemvumo, nezvimwe. Takatonyora kare , iyo inogonawo kuunganidzwa uye kuongororwa pano.
Pazasi pemepu yekupisa pane rondedzero yeanomalies, kubva kune yega yega iwe unogona chinja kune Imwe Metric maonero kuti uwane ongororo.
Population
Kutsvaga anomalies pakati pehukama pakati pemetrics akasiyana, iyo Elastic Stack ine yakasarudzika Population ongororo. Iri nerubatsiro rwayo kuti iwe unogona kutarisa maitiro asina kunaka mukuita kwesevha uchienzaniswa nevamwe apo, semuenzaniso, nhamba yezvikumbiro kune inotangwa system inowedzera.
Mumufananidzo uyu, iyo Population ndima inoratidza kukosha uko mametric akaongororwa achaenderana nawo. Muchiitiko ichi ndiro zita rekuita. Nekuda kweizvozvo, isu tichaona kuti iyo processor mutoro wega yega maitiro akapesvedzera mumwe nemumwe.
Ndokumbira utarise kuti girafu ye data yakaongororwa inosiyana nematare ane Single Metric uye Multi Metric. Izvi zvakaitwa muKibana nedhizaini yeruzivo rwakavandudzwa rwekugovera kukosha kweiyo data yakaongororwa.
Girafu inoratidza kuti maitiro akaita zvisina kujairika kushushikana (nenzira, inogadzirwa neakakosha utility) pane server poipu, uyo akafurira (kana kuti akazova mutsividzi) kuitika kweizvi zvisina kunaka.
enderera
Analytics nekugadzirisa kwakanaka. NeKuongorora Kwepamberi, mamwe marongero anoonekwa muKibana. Mushure mekudzvanya pane Yepamberi tile mune yekusika menyu, hwindo iri rine ma tabo rinoonekwa. Tab Job Job Isu takasvetuka nemaune, kune zvigadziriso zvekutanga zvisingaenderane nekugadzirisa ongororo.
Π summary_count_field_name Optionally, unogona kutsanangura zita remunda kubva kune zvinyorwa zvine aggregated values. Mumuenzaniso uyu, nhamba yezviitiko paminiti. IN inoratidza zita uye kukosha kwemunda kubva mugwaro rine humwe hukoshi hwakasiyana. Uchishandisa mask pane iyi ndima, unogona kupatsanura iyo yakaongororwa data kuita subsets. Nyatsoteerera bhatani Wedzera detector mumufananidzo wapfuura. Pazasi pane mhedzisiro yekudzvanya bhatani iri.
Heino imwe block yezvigadziriso zvekugadzirisa iyo anomaly detector yerimwe basa. Tinoronga kukurukura nyaya dzekushandisa (kunyanya dzekuchengetedza) munyaya dzinotevera. Semuyenzaniso, imwe yemakesi akaparadzaniswa. Iyo yakabatana nekutsvaga kweasingawanzo kuoneka kukosha uye inoshandiswa .
Mumunda basa Unogona kusarudza basa chairo kutsvaga anomalies. Kunze rova, kune mamwe maviri anonakidza mabasa - . Ivo vanocherekedza anomalies muhunhu hwemetrics zuva rose kana vhiki, zvichiteerana. Mamwe mabasa ekuongorora .
Π field_name inoratidza ndima yegwaro iro ongororo ichaitwa. By_field_name inogona kushandiswa kuparadzanisa mhinduro dzekuongorora kune yega yega kukosha kwegwaro remunda rataurwa pano. Ukazadza over_field_name unowana ongororo yehuwandu yatakurukura pamusoro. Kana iwe ukatsanangura kukosha mukati partition_field_name, zvino kune iyi ndima yegwaro mitsara yakapatsanurwa ichaverengerwa kukosha kwega kwega (kukosha kunogona kuve, semuenzaniso, zita revhavha kana maitiro pane server). IN bvisa_kazhinji inogona kusarudza zvose kana hapana, izvo zvinoreva kusasanganisa (kana kubatanidza) zvinowanzoitika magwaro emunda tsika.
Muchinyorwa chino, takaedza kupa sechipfupi zano sezvinobvira nezve kugona kwemuchina kudzidza muElastic Stack; kuchine zvakawanda zvakawanda zvakasiiwa kuseri kwezviitiko. Tiudze mumhinduro kuti ndedzipi nyaya dzawakakwanisa kugadzirisa uchishandisa Elastic Stack uye ndeapi mabasa aunoishandisa. Kuti ubate nesu, unogona kushandisa mameseji ega paHabrΓ© kana .
Source: www.habr.com