Masikhumbule ukuba i-Elastic Stack isekelwe kwi-database ye-Elasticsearch engeyiyo yobudlelwane, i-interface ye-Kibana yewebhu kunye nabaqokeleli bedatha kunye nabaqhubekisi (i-Logstash edume kakhulu, i-Beats eyahlukeneyo, i-APM kunye nabanye). Enye yezongezo ezintle kulo lonke uluhlu lwemveliso edwelisiweyo luhlalutyo lwedatha usebenzisa i-algorithms yokufunda koomatshini. Kwinqaku siyaqonda ukuba zeziphi ezi algorithms. Nceda phantsi kwekati.
Ukufunda ngomatshini luphawu oluhlawulwayo lwe-shareware Elastic Stack kwaye lufakwe kwi-X-Pack. Ukuqala ukuyisebenzisa, vele uvule uvavanyo lweentsuku ezingama-30 emva kofakelo. Emva kokuba ixesha lokuvavanya liphelile, unokucela inkxaso yokuyandisa okanye uthenge umrhumo. Ixabiso lokubhaliselwa libalwa lingasekelwe kumthamo wedatha, kodwa kwinani leendawo ezisetyenzisiweyo. Hayi, umthamo wedatha, ngokuqinisekileyo, uchaphazela inani lee-nodes ezifunekayo, kodwa kunjalo le ndlela yokufumana ilayisenisi inobuntu ngokumalunga nohlahlo lwabiwo-mali lwenkampani. Ukuba akukho mfuneko yemveliso ephezulu, unokonga imali.
I-ML kwi-Elastic Stack ibhalwe kwi-C++ kwaye isebenza ngaphandle kwe-JVM, apho i-Elasticsearch ngokwayo iqhuba khona. Oko kukuthi, inkqubo (ngendlela, ibizwa ngokuba yi-autodetect) idla yonke into engayigwinyi i-JVM. Kwidemo stand oku akubalulekanga kangako, kodwa kwimeko yemveliso kubalulekile ukwaba iindawo ezahlukeneyo zemisebenzi yeML.
Ii-algorithms zokufunda koomatshini ziwela kwiindidi ezimbini - ΠΈ . Kwi-Elastic Stack, i-algorithm ikwinqanaba "elingagadwanga". Ngu Uyakwazi ukubona izixhobo zemathematika ze-algorithms yokufunda koomatshini.
Ukwenza uhlalutyo, i-algorithm yokufunda umatshini isebenzisa idatha egcinwe kwii-indexes ze-Elasticsearch. Unokwenza imisebenzi yokuhlalutya zombini kwi-interface ye-Kibana kunye ne-API. Ukuba wenza oku ngeKibana, awudingi ukwazi ezinye izinto. Ngokomzekelo, izalathisi ezongezelelweyo ezisetyenziswa yi-algorithm ngexesha lokusebenza kwayo.
Izalathisi ezongezelelweyo ezisetyenziswa kwinkqubo yokuhlalutya.ml-state - ulwazi malunga neemodeli zamanani (uhlalutyo lwezicwangciso);
.ml-anomalies-* β iziphumo ze-ML algorithms;
.ml-izaziso - izicwangciso zezaziso ezisekelwe kwiziphumo zohlalutyo.
Ubume bedatha kwi-database ye-Elasticsearch iqukethe izalathisi kunye namaxwebhu agcinwe kuzo. Xa kuthelekiswa nesiseko sedatha yobudlelwane, isalathisi sinokuthelekiswa ne-schema yedatha, kunye noxwebhu kwirekhodi kwitafile. Olu thelekiso lunemiqathango kwaye lubonelelwe ukwenza lula ukuqonda kweminye imathiriyeli kwabo bathe beva kuphela nge-Elasticsearch.
Ukusebenza okufanayo kufumaneka nge-API njenge-interface yewebhu, ngoko ke ukucaca kunye nokuqonda iingcamango, siya kubonisa indlela yokuyiqwalasela nge-Kibana. Kwimenyu esekhohlo kukho icandelo leSifundo soomatshini apho unokwenza umsebenzi omtsha. Kwi-interface ye-Kibana ibonakala njengomfanekiso ongezantsi. Ngoku siza kuhlalutya uhlobo ngalunye lomsebenzi kwaye sibonise iintlobo zokuhlalutya ezinokwakhiwa apha.
I-Metric enye - uhlalutyo lwe-metric enye, i-Multi Metric - uhlalutyo lweemetriki ezimbini okanye ngaphezulu. Kuzo zombini iimeko, i-metric nganye ihlalutywa kwindawo esecaleni, okt. I-algorithm ayithatheli ngqalelo ukuziphatha kweemetrics ezihlalutyiweyo ezihambelanayo, njengoko kunokubonakala kwimeko ye-Multi Metric. Ukwenza izibalo ezithathela ingqalelo ulungelelwaniso lweemetrikhi ezahlukeneyo, ungasebenzisa uhlalutyo lwabemi. Kwaye i-Advanced ilungisa kakuhle i-algorithms kunye neenketho ezongezelelweyo zemisebenzi ethile.
I-Metric enye
Ukuhlalutya utshintsho kwimetric enye yeyona nto ilula enokwenziwa apha. Emva kokucofa kwi-Yenza umsebenzi, i-algorithm iya kukhangela i-anomalies.
Kwintsimi Abahlali ungakhetha indlela yokukhangela izinto ezingaqhelekanga. Ngokomzekelo, nini min amaxabiso angaphantsi kwamaxabiso aqhelekileyo aya kuthathwa njengento engaqhelekanga. Yitya Ubukhulu, buPhakamileyo, buPhantsi, buPhakathi, buPhakathi, obahlukileyo kunye nabanye. Iinkcazo zayo yonke imisebenzi zinokufumaneka .
Kwintsimi intsimi ibonisa indawo yamanani kuxwebhu esiya kuqhuba uhlalutyo.
Kwintsimi - i-granularity yezithuba kumgca wexesha apho uhlalutyo luya kwenziwa khona. Unokuthembela kwi-automation okanye ukhethe ngesandla. Lo mfanekiso ungezantsi ngumzekelo wobunzima obuphantsi kakhulu - ungaphoswa sisimanga. Ukusebenzisa olu cwangciso, unokutshintsha ubuntununtunu be-algorithm kwizinto ezingaqhelekanga.
Ubude bedatha eqokelelweyo yinto ephambili echaphazela ukusebenza kohlalutyo. Ngethuba lokuhlalutya, i-algorithm ichonga amaxesha aphindaphindiweyo, ibala amaxesha okuzithemba (isiseko) kwaye ichonge i-anomalies - ukuphambuka kwe-atypical ukusuka kwindlela yokuziphatha eqhelekileyo ye-metric. Umzekelo nje:
Iziseko ezineqhekeza elincinci ledatha:
Xa i-algorithm inento yokufunda kuyo, isiseko sijongeka ngolu hlobo:
Emva kokuqalisa umsebenzi, i-algorithm imisela ukutenxa ngendlela engaqhelekanga kwisiqhelo kwaye idwelise ngokuhambelana namathuba e-anomaly (umbala weleyibhile ehambelanayo uboniswe kwizibiyeli):
Isilumkiso (ebhlowu): ngaphantsi kwama-25
Encinci (etyheli): 25-50
Okukhulu (orenji): 50-75
Okubalulekileyo (obomvu): 75-100
Igrafu engezantsi ibonisa umzekelo wezinto ezingaqhelekanga ezifunyenweyo.
Apha ungabona inani lama-94, elibonisa ukuba nokwenzeka kwe-anomaly. Kucacile ukuba ekubeni ixabiso lisondele kwi-100, oko kuthetha ukuba sine-anomaly. Ikholamu engezantsi kwegrafu ibonisa ukuba nokwenzeka okuncinci kwe-0.000063634% yexabiso lemetriki elivela apho.
Ukongeza ekukhangeleni izinto ezingaqhelekanga, unokuqhuba uqikelelo eKibana. Oku kwenziwa ngokulula kwaye ukusuka kwimbono efanayo kunye ne-anomalies - iqhosha yaqikelelwa kwikona ephezulu ngasekunene.
Uqikelelo lwenzelwa ubuninzi beeveki ezisi-8 kwangaphambili. Nokuba ufuna ngokwenene, ayisenakwenzeka ngoyilo.
Kwezinye iimeko, uqikelelo luya kuba luncedo kakhulu, umzekelo, xa ubeka esweni umthwalo womsebenzisi kwiziseko zophuhliso.
Iimetriki ezininzi
Masiqhubele phambili kwinqaku elilandelayo leML kwiSitaki se-Elastic- sihlalutya iimetrikhi ezininzi kwibhetshi enye. Kodwa oku akuthethi ukuba ukuxhomekeka kwemetric enye kwenye kuya kuhlalutywa. Oku kuyafana neMetric enye, kodwa ngeemetrics ezininzi kwiscreen esinye ukuthelekisa ngokulula impembelelo enye kwenye. Siza kuthetha malunga nokuhlalutya ukuxhomekeka kwemetric enye kwenye kwicandelo labemi.
Emva kokunqakraza kwisikwere nge-Multi Metric, iwindow enezicwangciso iya kuvela. Makhe sizijonge ngakumbi.
Okokuqala kufuneka ukhethe iindawo zokuhlalutya kunye nokuhlanganiswa kwedatha kuzo. Iinketho zodibaniso apha ziyafana nezeMetric enye (Ubukhulu, buPhakamileyo, buPhantsi, buPhakathi, buPhakathi, obahlukileyo kunye nabanye). Ngaphaya koko, ukuba uyanqweneleka, idatha yahlulwe yaba yenye yemimandla (indawo Yahlula Data). Kumzekelo, senze oku ngentsimi I-OriginAirportID. Qaphela ukuba igrafu yeemetrics esekunene ngoku inikwa njengegrafu ezininzi.
Intsimi Imimandla engundoqo (Abaphembeleli) ichaphazela ngqo iziphazamiso ezifunyenweyo. Ngokungagqibekanga kuyakuhlala kukho nokuba linye ixabiso apha, kwaye unokongeza ezongezelelweyo. I-algorithm iya kuthathela ingqalelo impembelelo yale mimandla xa ihlalutya kwaye ibonise awona maxabiso "anamandla".
Emva kokuqaliswa, into enje iya kuvela kwi-interface ye-Kibana.
Oku kubizwa imephu yobushushu ye-anomalies yexabiso ngalinye lendawo I-OriginAirportID, esibonise kuyo Yahlula Data. NjengakwiMetric enye, umbala ubonisa inqanaba lokutenxa okungaqhelekanga. Kukulungele ukwenza uhlalutyo olufanayo, umzekelo, kwiindawo zokusebenza ukulandelela abo banenani elikhulu elikrokrelayo logunyaziso, njl. Besele sibhalile , enokuthi iqokelelwe kwaye ihlalutywe apha.
Ngaphantsi kwemephu yobushushu kukho uluhlu lwezinto ezididayo, ukusuka kwindawo nganye ungatshintshela kumbono weMetric enye ukuze uhlalutye oluneenkcukacha.
Lwabantu
Ukukhangela izinto ezingaqhelekanga phakathi kolungelelwaniso phakathi kweemetrics ezahlukeneyo, i-Elastic Stack inohlalutyo olukhethekileyo lwabemi. Kungoncedo lwayo ukuba unokukhangela amaxabiso angaqhelekanga ekusebenzeni komncedisi xa kuthelekiswa nabanye xa, umzekelo, inani lezicelo kwinkqubo ekujoliswe kuyo linyuka.
Kulo mzekeliso, indawo yePopulation ibonisa ixabiso apho i-metrics ehlaziyiweyo iya kunxulumana khona. Kule meko ligama lenkqubo. Ngenxa yoko, siza kubona ukuba umthwalo weprosesa wenkqubo nganye uphembelele njani enye kwenye.
Nceda uqaphele ukuba igrafu yedatha ehlalutyiweyo ihluke kwiimeko ezine-Single Metric kunye ne-Multi Metric. Oku kwenziwa e-Kibana ngoyilo lombono ophuculweyo wokusasazwa kwamaxabiso edatha ehlalutyiweyo.
Igrafu ibonisa ukuba inkqubo iziphathe ngendlela engaqhelekanga uxinzelelo (ngendlela, yenziwe yinto ekhethekileyo) kumncedisi ipopi, othe waphembelela (okanye wabonakala engumntu ophembelelayo) ukwenzeka koku kungaqhelekanga.
Advanced
Uhlalutyo olunohlengahlengiso olucokisekileyo. Ngohlalutyo oluPhezulu, izicwangciso ezongezelelweyo zivela kwi-Kibana. Emva kokunqakraza kwiTayile eNtu kwimenyu yokudala, le festile eneethebhu ibonakala. Ithebhu Iinkcukacha zeJob Siyigqithise ngenjongo, kukho izicwangciso ezisisiseko ezingahambelani ngokuthe ngqo nokuseta uhlalutyo.
Π summary_count_field_name Ngokhetho, ungakhankanya igama lendawo kumaxwebhu aqulathe amaxabiso adityanisiweyo. Kulo mzekelo, inani leziganeko ngomzuzu. IN ibonisa igama kunye nexabiso lomhlaba kuxwebhu oluqulathe ixabiso eliguquguqukayo. Ukusebenzisa imaski kule ndawo, unokwahlula idatha ehlalutyiweyo kwiiseti. Nika ingqalelo kwiqhosha Yongeza isixhobo kumzekeliso odlulileyo. Ngezantsi isiphumo sokucofa eli qhosha.
Nantsi ibhloko eyongezelelweyo yezicwangciso zokuqwalasela i-anomaly detector yomsebenzi othile. Siceba ukuxubusha iimeko ezithile zokusetyenziswa (ingakumbi ezokhuseleko) kumanqaku alandelayo. Umzekelo, enye yamatyala aqhawulweyo. Inxulunyaniswa nokukhangela amaxabiso anqabileyo kwaye aphunyeziwe .
Kwintsimi umsebenzi Unokukhetha umsebenzi othile ukukhangela izinto ezingaqhelekanga. Ngaphandle nqabile, kukho isibini semisebenzi enomdla ngakumbi - . Bachonga izinto ezingaqhelekanga ekuziphatheni kweemetrics imini yonke okanye iveki, ngokulandelelanayo. Eminye imisebenzi yohlalutyo .
Π indawo_igama ibonisa intsimi yoxwebhu apho uhlalutyo luya kwenziwa khona. Nge_indawo_igama ingasetyenziselwa ukwahlula iziphumo zohlalutyo lwexabiso lomntu ngamnye wendawo yoxwebhu echazwe apha. Ukuba ugcwalise ngaphezulu_kwendawo_igama ufumana uhlalutyo lwabemi esiluxoxe ngasentla. Ukuba ukhankanya ixabiso kwi isahlulo_indawo_igama, ngoko kulo mhlaba woxwebhu iziseko ezisisiseko ezahlukileyo ziya kubalwa kwixabiso ngalinye (ixabiso lingaba, umzekelo, igama lomncedisi okanye inkqubo kumncedisi). IN ngaphandle_kaninzi ungakhetha zonke okanye nanye, okuya kuthetha ukungaquki (okanye ukuquka) amaxabiso omhlaba ahlala esenzeka rhoqo.
Kweli nqaku, sizame ukunika ingcamango ecacileyo ngokusemandleni okufunda koomatshini kwi-Elastic Stack kusekho iinkcukacha ezininzi ezishiywe ngasemva. Sixelele kwizimvo ukuba zeziphi iimeko oye wakwazi ukuzicombulula usebenzisa i-Elastic Stack kwaye yeyiphi imisebenzi oyisebenzisela yona. Ukuqhagamshelana nathi, ungasebenzisa imiyalezo yobuqu kwiHabrΓ© okanye .
umthombo: www.habr.com