{"id":35274,"date":"2019-10-31T22:03:22","date_gmt":"2019-10-31T19:03:22","guid":{"rendered":"https:\/\/prohoster.info\/blog\/razbiraemsya-s-machine-learning-v-elastic-stack-on-zhe-elasticsearch-on-zhe-elk\/"},"modified":"2019-10-31T22:03:22","modified_gmt":"2019-10-31T19:03:22","slug":"razbiraemsya-s-machine-learning-v-elastic-stack-on-zhe-elasticsearch-on-zhe-elk","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razbiraemsya-s-machine-learning-v-elastic-stack-on-zhe-elasticsearch-on-zhe-elk","title":{"rendered":"Understanding Machine Learning in the Elastic Stack (also known as Elasticsearch, also ELK)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Understanding
\n
\nLet's recall that the Elastic Stack is based on the non-relational database Elasticsearch, the web interface Kibana, and data processors (the most famous being Logstash, various Beats, APM, and others). One of the nice additions to this product stack is data analysis using machine learning algorithms. In this article, we will explore what these algorithms are all about. We invite you to read on.
\n<\/a><\/noindex>
\nMachine learning is a paid feature of the somewhat free Elastic Stack and is included in the X-Pack package. To start using it, you need to activate a 30-day trial after installation. Once the trial period ends, you can request support for an extension or purchase a subscription. The subscription price is based not on the volume of data but rather on the number of nodes used. While the data volume does influence the number of required nodes, this licensing approach is more budget-friendly for companies. If high performance is not necessary, you can save costs.<\/p>\n

ML in the Elastic Stack is written in C++ and operates outside the JVM, in which Elasticsearch itself runs. This means the process (which is called autodetect, by the way) consumes everything that the JVM does not handle. This is not critical on a demo stand, but in a production environment, it's important to allocate separate nodes for ML tasks.<\/p>\n

Machine learning algorithms are divided into two categories \u2014 with a teacher<\/a><\/noindex> and without a teacher<\/a><\/noindex>. In the Elastic Stack, the algorithm falls into the 'without a teacher' category. You can refer to this link<\/a><\/noindex> to see the mathematical framework of machine learning algorithms.<\/p>\n

For conducting analysis, the machine learning algorithm uses data stored in Elasticsearch indices. Jobs for analysis can be created from both the Kibana interface and through the API. If doing this via Kibana, some knowledge of certain things is not strictly necessary. For example, the additional indices that the algorithm uses during its operation. <\/p>\n

Additional indices used during the analysis process<\/b>.ml-state \u2014 information about statistical models (analysis settings);
\n.ml-anomalies-* \u2014 results from the operation of the ML algorithms;
\n.ml-notifications \u2014 notification settings based on analysis results.<\/p>\n

\"Understanding
\n
\nThe data structure in an Elasticsearch database consists of indices and the documents stored within them. If we compare this to a relational database, an index can be likened to a database schema, while a document is akin to a record in a table. This comparison is conditional and is provided to simplify understanding of further material for those who have only heard about Elasticsearch.<\/p>\n

The same functionality that is available through the web interface is accessible via the API; therefore, for clarity and understanding of the concepts, we will show how to configure it through Kibana. On the left menu, there is a section called Machine Learning, where you can create a new job. In the Kibana interface, this looks like the image below. We will now break down each type of job and demonstrate the types of analysis that can be constructed here.<\/p>\n

\"Understanding
\n
\nSingle Metric \u2014 analysis of a single metric, Multi Metric \u2014 analysis of two or more metrics. In both cases, each metric is analyzed in isolation, meaning the algorithm does not account for the behavior of concurrently analyzed metrics, as might appear in the case of Multi Metric. To conduct calculations considering the correlation of various metrics, one can apply Population analysis. Advanced analysis is a fine-tuning of algorithms with additional options for specific tasks. <\/p>\n

Single Metric<\/h2>\n

\nAnalyzing changes in a single metric is the simplest task that can be performed here. After clicking Create Job, the algorithm will search for anomalies.<\/p>\n

\"Understanding
\n
\nIn the field Aggregation<\/i> you can choose the approach to anomaly detection. For example, with Min<\/i> values below typical levels will be considered anomalous. There are also Max, High Mean, Low, Mean, Distinct<\/i> and others. A description of all functions can be found at the link<\/a><\/noindex>.<\/p>\n

In the field Field<\/i> specifies the numeric field in the document by which we will conduct the analysis.<\/p>\n

In the field Bucket span<\/i><\/a><\/noindex> is the granularity of intervals on the timeline for the analysis. You can trust the automation or select manually. The image below shows an example of too low granularity \u2014 you may miss an anomaly. This setting can adjust the algorithm's sensitivity to anomalies.<\/p>\n

\"Understanding
\n
\nThe duration of collected data is a key factor that influences the effectiveness of the analysis. In the analysis, the algorithm identifies repeating intervals, calculates the confidence interval (baseline), and detects anomalies \u2014 atypical deviations from the usual behavior of the metric. Just for example:<\/p>\n

Baseline for small data segments:<\/p>\n

\"Understanding
\n
\nWhen the algorithm has something to learn from, the baseline looks like this:<\/p>\n

\"Understanding
\n
\nAfter the task is launched, the algorithm identifies anomalous deviations from the norm and ranks them by anomaly probability (the color of the corresponding label is indicated in parentheses):<\/p>\n

Warning (blue): less than 25
\nMinor (yellow): 25-50
\nMajor (orange): 50-75
\nCritical (red): 75-100<\/p>\n

The graph below shows an example with identified anomalies.<\/p>\n

\"Understanding
\n
\nHere we see the number 94, which indicates the probability of an anomaly. It\u2019s clear that since the value is close to 100, this is an anomaly. The column under the graph indicates a negligible probability of 0.000063634% of the metric value appearing there.<\/p>\n

In addition to anomaly detection in Kibana, you can run forecasting. This is done easily from the same view with anomalies \u2014 click the Forecast<\/i> button in the upper right corner.<\/p>\n

\"Understanding
\n
\nThe forecast is built a maximum of 8 weeks ahead. Even if you wish for more, it's not possible by design.<\/p>\n

\"Understanding
\n
\nIn some cases, forecasting will be very useful, for example, when monitoring user load on infrastructure.<\/p>\n

Multi Metric<\/h2>\n

\nNext, we move to another ML capability in the Elastic Stack \u2014 analyzing multiple metrics as a batch. However, this does not mean that the dependence of one metric on another will be analyzed. It's the same as Single Metric, just with multiple metrics on one screen for easy comparison of their influence on each other. We will discuss the analysis of the dependence of one metric on another in the Population section.<\/p>\n

After clicking on the square with Multi Metric, a settings window will appear. Let's take a closer look at these settings.<\/p>\n

\"Understanding
\n
\nFirst, you need to select fields for analysis and aggregate data on them. The aggregation options here are the same as for Single Metric (Max, High Mean, Low, Mean, Distinct<\/i> and others). Next, data can be split by one of the fields if desired (field Split Data<\/i>). In the example, we did this by the field OriginAirportID<\/i>. Note that the metrics graph on the right is now presented as multiple graphs.<\/p>\n

\"Understanding
\n
\nField Key Fields (Influencers)<\/i> directly affect the identified anomalies. By default, there will always be at least one value here, and you can add more. The algorithm will take these fields into account during analysis and show the most \"influential\" values.<\/p>\n

After running, the interface in Kibana will look something like this.<\/p>\n

\"Understanding
\n
\nThis is the so-called heatmap of anomalies for each value of the field OriginAirportID<\/i>, which we specified in Split Data<\/i>As with Single Metric, the color indicates the level of anomalous deviation. A similar analysis can conveniently be performed, for example, on workstations to monitor those with suspiciously high logins, etc. We have previously written about suspicious events in EventLog Windows<\/a><\/noindex>, which can also be collected and analyzed here.<\/p>\n

Under the heat map, there is a list of anomalies, each of which can be clicked to access the Single Metric view for detailed analysis.<\/p>\n

Population<\/h2>\n

\nTo search for anomalies among correlations between different metrics in Elastic Stack, there is a specialized Population analysis. It is precisely through this that anomalous values in the performance of a server can be explored in relation to others when, for example, the number of requests to the target system increases.<\/p>\n

\"Understanding
\n
\nIn this illustration, the Population field shows the value to which the analyzed metrics will relate. In this case, it is the process name. As a result, we will see how the CPU load of each of the processes influenced each other.<\/p>\n

Note that the graph of the analyzed data differs from those with Single Metric and Multi Metric. This is done in Kibana by design for improved perception of the distribution of the analyzed data values.<\/p>\n

\"Understanding
\n
\nThe graph shows that the process behaved anomalously stress<\/i> (by the way, generated by a special utility) on the server poipu<\/i>, which influenced (or was an influencer of) the emergence of this anomaly.<\/p>\n

Advanced<\/h2>\n

\nAnalytics with fine-tuning. When performing an Advanced analysis in Kibana, additional settings appear. After clicking on the creation menu on the Advanced tile, a window with tabs appears. The tab Job Details<\/i> was intentionally skipped; it contains basic settings not directly related to the analysis configuration.<\/p>\n

\"Understanding
\n
\nIn summary_count_field_name<\/i> optionally allows specifying the name of the field from the documents that contains aggregated values. In this example \u2014 the number of events per minute. In the categorization_field_name<\/i><\/a><\/noindex> , the name of the field from the document that contains some variable value is specified. This field can be used as a mask to break down the analyzed data into subsets. Note the button Add detector<\/i> in the previous illustration. Below is the result of clicking this button.<\/p>\n

\"Understanding
\n
\nHere is an additional settings block for configuring the anomaly detector for a specific task. We plan to discuss specific use cases (especially related to security) in upcoming articles. For example, check out<\/a><\/noindex> one of the discussed cases. It relates to finding rarely occurring values and is implemented using the rare function<\/a><\/noindex>.<\/p>\n

In the field function<\/i> you can choose a specific function for anomaly detection. Besides rare<\/i>, there are a couple of interesting functions \u2014 time_of_day<\/i> and time_of_week<\/i><\/a><\/noindex>. They uncover anomalies in metric behavior over the course of a day or week, respectively. Other analysis functions are in the documentation<\/a><\/noindex>.<\/p>\n

In field_name<\/i> specifies the field of the document to be analyzed. By_field_name<\/i> can be used to split the analysis results by each individual value of the specified document field. If you fill in over_field_name<\/i> you will get a population analysis, which we discussed earlier. If a value is specified in partition_field_name<\/i>, separate baselines will be calculated for each value based on this document field (for example, the server name or the process name on the server can act as a value). In exclude_frequent<\/i> you can select all<\/i> or none<\/i>, which will mean excluding (or including) frequently occurring document field values.<\/p>\n

In this article, we tried to provide a concise overview of machine learning capabilities in the Elastic Stack, leaving out many details. Let us know in the comments what cases you have solved using the Elastic Stack and for what tasks you use it. To contact us, you can use direct messages on Habr or the feedback form on the website<\/a><\/noindex>.
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041d\u0430\u043f\u043e\u043c\u043d\u0438\u043c, \u0447\u0442\u043e \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 Elastic Stack \u043b\u0435\u0436\u0430\u0442 \u043d\u0435\u0440\u0435\u043b\u044f\u0446\u0438\u043e\u043d\u043d\u0430\u044f \u0431\u0430\u0437\u0430 \u0434\u0430\u043d\u043d\u044b\u0445 Elasticsearch, \u0432\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 Kibana \u0438 \u0441\u0431\u043e\u0440\u0449\u0438\u043a\u0438-\u043e\u0431\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 (\u0441\u0430\u043c\u044b\u0439 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u044b\u0439 Logstash, \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0435 Beats, APM \u0438 \u0434\u0440\u0443\u0433\u0438\u0435). \u041e\u0434\u043d\u043e \u0438\u0437 \u043f\u0440\u0438\u044f\u0442\u043d\u044b\u0445 \u0434\u043e\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u0439 \u0432\u0441\u0435\u0433\u043e \u043f\u0435\u0440\u0435\u0447\u0438\u0441\u043b\u0435\u043d\u043d\u043e\u0433\u043e \u0441\u0442\u0435\u043a\u0430 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432 \u2014 \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445 \u043f\u0440\u0438 \u043f\u043e\u043c\u043e\u0449\u0438 \u0430\u043b\u0433\u043e\u0440\u0438\u0442\u043c\u043e\u0432 \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f. \u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0430\u0437\u0431\u0438\u0440\u0430\u0435\u043c\u0441\u044f \u0447\u0442\u043e \u0438\u0437 \u0441\u0435\u0431\u044f \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e\u0442 \u044d\u0442\u0438 \u0430\u043b\u0433\u043e\u0440\u0438\u0442\u043c\u044b. \u041f\u0440\u043e\u0441\u0438\u043c \u043f\u043e\u0434 \u043a\u0430\u0442. \u041c\u0430\u0448\u0438\u043d\u043d\u043e\u0435 \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26487,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-35274","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Exploring Machine Learning in the Elastic Stack (also known as Elasticsearch, or ELK) | ProHoster","description":"Let's remind you.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razbiraemsya-s-machine-learning-v-elastic-stack-on-zhe-elasticsearch-on-zhe-elk","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0420\u0430\u0437\u0431\u0438\u0440\u0430\u0435\u043c\u0441\u044f \u0441 Machine Learning \u0432 Elastic Stack (\u043e\u043d \u0436\u0435 Elasticsearch, \u043e\u043d \u0436\u0435 ELK) | ProHoster","og:description":"\u041d\u0430\u043f\u043e\u043c\u043d\u0438\u043c.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razbiraemsya-s-machine-learning-v-elastic-stack-on-zhe-elasticsearch-on-zhe-elk","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:03:22+00:00","article:modified_time":"2019-10-31T19:03:22+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35274","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 22:37:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:07:28","updated":"2026-01-21 22:37:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35274","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=35274"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35274\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/26487"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=35274"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=35274"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=35274"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}