In We talked about time series forecasting. A logical continuation would be an article on anomaly detection.
The use of
Anomaly detection is used in areas such as:
1) Predicting equipment failures
For instance, in 2010, Iranian centrifuges were attacked by the Stuxnet virus, which set the equipment to an optimal operating mode and caused part of it to fail due to accelerated wear.
If anomaly detection algorithms were used on the equipment, these failures could have been avoided.
Anomaly detection in equipment operation is used not only in the nuclear industry but also in metallurgy and aviation turbine operation, and in other areas where predictive diagnostics are cheaper than the potential losses from unforeseen failures.
2) Predicting fraudulent activities
If money is withdrawn from a card you use in Podolsk while in Albania, it may be wise to verify the transactions.
3) Identifying anomalous consumer patterns
If a portion of customers is demonstrating abnormal behavior, there may be an issue that you are unaware of.
4) Detecting abnormal demand and workload
If sales in an FMCG store drop below the lower limit of the confidence interval forecast, it is worth finding the cause of the situation.
Approaches to anomaly detection
1) One-Class SVM
Suitable when the training set data follow a normal distribution, while the test data contain anomalies.
The one-class support vector method constructs a nonlinear surface around the origin. It allows setting a cutoff boundary to determine what data is considered anomalous.
Based on the experience of our DATA4 team, One-Class SVM is the most frequently used algorithm for solving the anomaly detection task.
2) Isolation Forest
With a 'random' method of tree construction, outliers will end up in leaves at early stages (with a small tree depth), i.e., outliers are easier to 'isolate'. Anomalous values are identified in the early iterations of the algorithm's operation.
3) Elliptic Envelope and statistical methods
Used when data is normally distributed. The closer the measurement is to the tails of the distribution mixture, the more anomalous the value.
This class can also include other statistical methods.
Image from dyakonov.org
4) Metric methods
The methods include algorithms such as k-nearest neighbors, k-th nearest neighbor, ABOD (angle-based outlier detection), or LOF (local outlier factor).
They are suitable if the distances between values in the features are equivalent or normalized (to avoid measuring a python in parrots).
The k-nearest neighbors algorithm assumes that normal values are located within a certain area of the multidimensional space, and the distance to anomalies will be greater than to the separating hyperplane.
5) Cluster methods
The essence of cluster methods is that if a value is far from the cluster centers by more than a certain amount, it can be considered an anomaly.
The main thing is to use an algorithm that properly clusters the data, which depends on the specific task.
6) Principal component method
Suitable where directions of greatest variance are highlighted.
7) Time series forecasting algorithms
The idea is that if a value falls outside the confidence interval of the prediction, it is considered an anomaly. Algorithms such as triple smoothing, S(ARIMA), boosting, etc., are used for time series forecasting.
The time series forecasting algorithms were discussed in the previous article.
8) Supervised learning (regression, classification)
If the data allows, we use algorithms ranging from linear regression to recurrent networks. We will measure the difference between the prediction and the actual value, and conclude how much the data deviates from the norm. It is important that the algorithm has sufficient generalization ability and the training sample does not contain anomalies.
9) Model tests
We will approach the task of anomaly detection as a recommendation problem. We will decompose our feature matrix using SVD or factorization machines, and values in the new matrix that significantly differ from the original will be accepted as anomalous.
Image from dyakonov.org
Conclusion
In this article, we reviewed the main approaches to anomaly detection.
Anomaly detection can largely be considered an art. There is no perfect algorithm or approach that solves all tasks. A combination of methods is often used to tackle specific cases. Anomaly detection is carried out using the one-class support vector method, isolation forests, metric and clustering methods, as well as principal component analysis and time series forecasting.
If you know other methods, please share them in the comments section of the article.
Source: habr.com
