Hello, Habr!
At 'Rexoft', we have translated an article into Russian . We hope it will be useful for everyone interested in the topic.
In the real world, data is not always as clean as business stakeholders often think. This is why there is a demand for data mining and data wrangling. It helps to identify missing values and patterns in structured data through queries that cannot be determined by humans. To find and utilize these patterns for predicting outcomes using the discovered relationships in the data, machine learning will come in handy.
To understand any algorithm, it's essential to examine all variables in the data and figure out what these variables represent. This is crucial because the justification of results relies on understanding the data. If the data contains 5 or even 50 variables, one can study all of them. But what if there are 200? Then, there simply isn't enough time to examine each individual variable. Moreover, some algorithms do not work with categorical data; in that case, it will be necessary to convert all categorical columns into quantitative variables (they may appear quantitative, but the metrics will show that they are categorical) in order to include them in the model. Therefore, the number of variables increases, and it becomes around 500. What now? One might think that the solution would be dimensionality reduction. Dimensionality reduction algorithms decrease the number of parameters but negatively impact interpretability. What if there are other techniques that exclude features while allowing for easy understanding and interpretation of the remaining ones?
Depending on whether the analysis is based on regression or classification, feature selection algorithms may differ, but the main idea of their implementation remains the same.
Highly correlated variables
Highly correlated variables provide the model with the same information; therefore, it is not necessary to use all of them for analysis. For example, if a dataset contains features like 'Time Online' and 'Data Used', it can be assumed that they will be somewhat correlated, and we will see a strong correlation even if we select an unbiased sample of data. In this case, only one of these variables is needed in the model. Using both would lead to overfitting and bias towards one specific feature.
P-values
In algorithms like linear regression, starting with a statistical model is always a good idea. It helps illustrate the importance of features through their p-values obtained by this model. By setting a significance level, we check the obtained p-values, and if any value falls below the specified significance level, that feature is deemed significant, meaning that a change in its value will likely lead to a change in the target value.
Forward Selection
Forward selection is a technique that involves stepwise regression. The model building starts from scratch, that is, an empty model, and then each iteration adds a variable that improves the model being built. The decision on which variable to add is determined by its significance. This can be calculated using various metrics, with the most common method being the application of p-values obtained from the initial statistical model using all variables. Sometimes forward selection can lead to model overfitting because highly correlated variables may end up in the model, even if they provide the same information, while the model still shows improvement.
Backward Elimination
Reverse selection also involves the step-by-step exclusion of features, but in the opposite direction compared to forward selection. In this case, the initial model includes all independent variables. Then, variables are excluded (one at a time per iteration) if they do not provide value to the new regression model at each iteration. The basis for excluding features relies on the p-value metrics of the initial model. This method also carries uncertainty when removing highly correlated variables.
Recursive Feature Elimination
RFE is a widely used technique/algorithm for selecting the precise number of significant features. Sometimes the method is used to explain a certain number of the 'most important' features affecting the outcomes; other times, it is used to reduce a very large number of variables (about 200-400), leaving only those that contribute even a little to the model, while all others are excluded. RFE employs a ranking system. Features in the dataset are assigned ranks. These ranks are then used for recursive feature elimination based on the collinearity among them and the significance of these features in the model. In addition to ranking features, RFE can indicate whether these features are important or not, even for a given number of features (because it is very likely that the chosen number of features may not be optimal, and the optimal number may be either greater or less than the selected number).
Feature Importance Chart
When discussing the interpretability of machine learning algorithms, linear regressions (which allow analyzing feature importance using p-values) and decision trees (which literally show feature importance in the form of a tree, as well as their hierarchy) are usually discussed. On the other hand, in algorithms such as Random Forest, LightGBM, and XGBoost, a feature importance chart is often used, meaning a chart of variables and their 'importance scores' is constructed. This is especially useful when needing to provide a structured justification for feature importance in terms of their impact on business.
Regularization
Regularization is used to control the balance between bias and variance. Bias indicates how much the model is overfitting to the training dataset. Variance shows how different the predictions were between the training and test datasets. Ideally, both bias and variance should be low. This is where regularization comes in! There are two main techniques:
L1 Regularization — Lasso: Lasso penalizes the model's weight coefficients to alter their importance and can even set them to zero (i.e., removing these variables from the final model). Lasso is typically used when the dataset contains a large number of variables and some need to be excluded to better understand how significant features impact the model (i.e., the features selected by Lasso that have importance assigned).
L2 Regularization — Ridge: The goal of Ridge is to retain all variables while assigning them importance based on their contribution to model performance. Ridge is a good choice when the dataset has a small number of variables and all of them are necessary for interpreting the conclusions and results obtained.
Since Ridge keeps all variables while Lasso does a better job of determining their importance, an algorithm combining the best features of both regularizations was developed, known as Elastic-Net.
There are many other feature selection methods for machine learning, but the main idea always remains the same: to demonstrate the importance of variables and then exclude some based on the importance obtained. Importance is a very subjective term, as it is not just one but a whole set of metrics and charts that can be used for identifying key features.
Thank you for reading! Happy learning!
Source: habr.com
