🥇How to package a VueJS + NodeJS + MongoDB application in Docker | ProHoster

Inspired by my presentations at Highload++ and DataFest Minsk 2019.

For many today, email is an integral part of life online. Through it, we conduct business correspondence, store all kinds of important information related to finances, hotel bookings, order placements, and much more. In mid-2018, we formulated a product strategy for the development of email. What should modern email look like?

Email must be smart, meaning it should help users navigate the increasing volume of information: filter, structure, and present it in the most convenient way. It should be beneficial, allowing users to solve various tasks directly from their inbox, for instance, paying fines (a feature I, unfortunately, find myself using). And of course, it must ensure information security, filtering out spam and protecting against hacks, meaning it should be secure.

These directions define a number of key tasks, many of which can be effectively addressed using machine learning. Here are examples of features already in place, developed as part of the strategy — one for each direction.

• Smart Reply. The email has a Smart Reply feature. The neural network analyzes the text of the email, understands its meaning and purpose, and consequently offers three most suitable response options: positive, negative, and neutral. This significantly saves time when responding to emails and often helps in responding in a unique and amusing way.
• Grouping emails, related to orders from online stores. We often shop online, and typically, stores can send several emails for each order. For example, from AliExpress, the largest service, a lot of emails can come for a single order, and we found that in extreme cases, their number can reach up to 29. Therefore, using the Named Entity Recognition model, we extract the order number and other information from the text and group all emails into one thread. We also display the main information about the order in a separate panel, making it easier to manage this type of emails.

  

• Anti-PhishingPhishing is a particularly dangerous type of fraudulent email in which attackers try to gain control of financial information (including users' credit card details) and logins. Such emails mimic real ones sent by legitimate services, including visually. Therefore, with the help of Computer Vision, we recognize the logos and design styles of major companies' emails (for example, Mail.ru, Sber, Alfa) and take this into account along with text and other features in our spam and phishing classifiers.

Machine Learning

A bit about machine learning in mail overall. Mail is a high-load system: an average of 1.5 billion emails pass through our servers daily for 30 million daily active users. Approximately 30 machine learning systems handle all the necessary functions and features.

Every email goes through a complete classification pipeline. First, we filter out spam and keep good emails. Users often do not notice the work of the anti-spam system because 95-99% of spam doesn't even reach the spam folder. Spam recognition is a very important part of our system and one of the most complex, as there is constant adaptation between protection systems and attacks, presenting a continuous engineering challenge for our team.

Next, we separate emails from people and bots. Emails from people are the most important, so we provide features like Smart Reply for them. Emails from bots are divided into two parts: transactional — important emails from services, such as purchase confirmations or hotel bookings, finances, and informational — business promotions, discounts.

We believe that transactional emails are as significant as personal correspondence. They should be readily available, as it’s often necessary to quickly find information about an order or a flight booking, and we waste time searching for these emails. Therefore, for convenience, we automatically sort them into six main categories: travel, orders, finances, tickets, registrations, and finally, fines.

Information letters are the most numerous and probably the least important group, not requiring immediate action, as nothing significant will change in the user's life if they do not read such a letter. In our new interface, we group them into two threads: social media and newsletters, thus visually clearing the inbox and displaying only important emails.

Operation

A large number of systems brings a lot of difficulties in operation. Models degrade over time just like any software: features break, machines fail, and buggy code accumulates. Additionally, data is constantly changing: new data is added, user behavior patterns transform, etc., so a model without proper support will gradually perform worse and worse over time.

It is also important to remember that the deeper machine learning penetrates users' lives, the greater influence they have on the ecosystem. Consequently, market players can incur greater financial losses or profits. Therefore, in more areas, players are adapting to work with ML algorithms (classic examples include advertising, search, and the previously mentioned anti-spam).

Moreover, machine learning tasks have an important feature: any, even minor, change in the system can generate a lot of work with the model: data handling, retraining, deployment, which can take weeks or months. Therefore, the faster the environment in which your models operate changes, the more effort is required to maintain them. A team can create many systems and be pleased with this, but then spend almost all resources on their maintenance without the opportunity to develop anything new. We encountered such a situation once in the anti-spam team and came to the obvious conclusion that maintenance needs to be automated.

Automation

What can be automated? In fact, almost everything. I have identified four areas that define the machine learning infrastructure:

• data collection;
• retraining;
• deployment;
• testing & monitoring.

If the environment is unstable and constantly changing, then the entire infrastructure surrounding the model becomes much more important than the model itself. It could be a good old linear classifier, but if the features are fed into it correctly and good feedback from users is established, it will perform far better than State-Of-The-Art models with all the frills.

Feedback Loop

This loop encompasses data collection, retraining, and deployment — essentially, the entire model update cycle. Why is this important? Look at the registration chart of the mail system:

The machine learning developer implemented an anti-bot model that prevents bots from registering in the mail system. The chart drops to a level where only real users remain. Everything looks great! But after four hours, spammers tweak their scripts, and everything goes back to square one. In this implementation, the developer spent a month adding features and retraining the model, but the spammer was able to adapt in just four hours.

To avoid painful setbacks and the need to redo everything later, it’s essential to think initially about how the feedback loop will look and what we will do if the environment changes. Let’s start with data collection — it’s the fuel for our algorithms.

Data Collection

It is clear that for modern neural networks, the more data, the better, and this data is essentially generated by the product's users. Users can help by annotating the data, but this cannot be overdone, as users will eventually get tired of retraining your models and will switch to another product.

One of the most common mistakes (here, I'm referencing Andrew Ng) is being overly focused on metrics on the test dataset rather than on user feedback, which is actually the main measure of quality since we are creating a product for users. If the user finds the model's performance unclear or unsatisfactory, that means everything is in vain.

Therefore, users must always have the ability to vote; we should provide them with a tool for feedback. If we consider that an email in the inbox relates to finances, we need to tag it as 'finance' and create a button that the user can click to indicate that it is not finances.

Quality of Feedback

Let's discuss the quality of user feedback. Firstly, you and the users may attach different meanings to the same concept. For instance, you and the product managers might consider "finance" as emails from the bank, while the user thinks that a letter from grandma about retirement also falls under finance. Secondly, there are users who thoughtlessly like to click buttons with no logic at all. Thirdly, a user may be deeply mistaken in their conclusions. A vivid example from our practice is the implementation of a classifying system for Nigerian spam, a rather amusing type of spam where users are offered to claim several million dollars from a suddenly found distant relative in Africa. After implementing this classifier, we checked the "Not Spam" clicks on these emails, and it turned out that 80% of them were juicy Nigerian spam, indicating that users can be extremely gullible.

And let's not forget that clicks can be made not only by people but also by various bots pretending to be browsers. Therefore, raw feedback is not suitable for training. What can we do with this information?

We apply two approaches:

• Feedback from related ML. For example, we have an online anti-bot system that, as I mentioned before, makes quick decisions based on a limited number of features. There is a second, slower system that works retrospectively. It has more data about the user, their behavior, etc. As a result, the most considered decision is made, thus it has higher accuracy and completeness. The difference between the workings of these systems can be fed into the first one as training data. This way, the simpler system will always strive to approximate the performance of the more complex one.
• Click classificationYou can simply classify each user click, assess its validity and usability. We do this in email anti-spam by using user traits, their history, sender characteristics, the content itself, and the classifiers' output. As a result, we get an automated system that validates user feedback. Since it needs to be retrained significantly less often, its performance can become fundamental for all other systems. The main priority in this model is precision, because training the model on inaccurate data can have serious consequences.

While we clean the data and retrain our ML systems, we must not forget about the users; for us, thousands or even millions of errors on a graph represent statistics, but for the user, each bug is a tragedy. Besides the fact that users have to deal with your product's error, they expect similar situations to be excluded in the future after providing feedback. Therefore, it's always worth giving users not just the chance to vote but also to correct the behavior of the ML systems, for example, by creating personal heuristics for each feedback click; in the case of email, this could mean the ability to filter such emails by sender and subject for that user.

It is also necessary to use some reports or support requests to semi-automatically or manually tweak the model so that other users do not suffer from similar problems.

Heuristics for training

There are two problems with the current heuristics and tweaks. The first is that the ever-increasing number of tweaks is difficult to maintain, not to mention their quality and performance over the long term. The second issue is that an error may not be frequent, and a few clicks to retrain the model may be insufficient. These two seemingly unrelated effects could be significantly mitigated by applying the following approach.

1. We create a temporary tweak.
2. We send data from it to the model, which is regularly retrained, including on the obtained data. Here, of course, it's important for the heuristic to have high accuracy in order not to degrade the quality of data in the training set.
3. Then we set up monitoring for the workaround, and if after some time the workaround no longer activates and is fully covered by the model, it can safely be removed. This problem is unlikely to recur now.

Thus, the army of workarounds is very useful. The main thing is that their service is urgent and not permanent.

Fine-tuning

Fine-tuning is the process of adding new data obtained from user feedback or other systems and training the existing model on it. There can be several issues with fine-tuning:

1. The model may simply not support fine-tuning and can only learn from scratch.
2. Nowhere in the book of nature is it written that fine-tuning will necessarily improve the quality of production performance. Often the opposite occurs, meaning it is possible only to worsen.
3. Changes can be unpredictable. This is a delicate point we have identified. Even if the new model shows similar results in A/B testing compared to the current one, it does not mean it will perform identically. Their performance may differ in some small percentage, which can introduce new errors or revive already fixed old ones. With current errors, both we and the users have learned to live, and when a large number of new errors arise, the user may not understand what is happening, as they expect predictable behavior.

Therefore, the most important thing in fine-tuning is to guarantee an improvement in the model, or at least not to worsen it.

The first thought that comes to mind when we talk about fine-tuning is the Active Learning approach. What does this mean? For example, a classifier determines whether an email relates to finances, and around its decision boundary, we add a sample of labeled examples. This works well, for instance, in advertising, where feedback is plentiful and the model can be trained in real-time. But if there is little feedback, we get a severely biased sample relative to the production data distribution, on which it is impossible to evaluate model behavior during operation.

Our goal is to preserve established patterns, known models, and acquire new ones. Continuity is important here. The model, which we have often rolled out with great difficulty, is already functioning, so we can rely on its performance.

Various models are applied in the mail: trees, linear models, neural networks. We create a specific fine-tuning algorithm for each one. During the fine-tuning process, we not only acquire new data but also often identify new features that we will consider in all subsequent algorithms.

Linear models

Let's take logistic regression as an example. We formulate the model's loss from the following components:

• LogLoss on new data;
• we regularize the weights of new features (we leave the old ones unchanged);
• we also learn on the old data to retain existing patterns;
• and perhaps most importantly: we apply Harmonic Regularization, which ensures the weights do not change significantly from the old model norm.

Since each component of the Loss has coefficients, we can find optimal values for our task through cross-validation or based on product requirements.

Trees

Let’s move on to decision trees. We have developed the following fine-tuning algorithm for trees:

1. In production, there’s a forest of 100–300 trees that has been trained on the old dataset.
2. At the end, we remove M = 5 trees and add 2M = 10 new ones, trained on the entire dataset but with a high weight on the new data, which naturally guarantees an incremental change in the model.

It’s evident that over time, the number of trees significantly increases, and they need to be periodically reduced to fit within timing constraints. To achieve this, we use the now-ubiquitous Knowledge Distillation (KD). Here’s a brief overview of how it works.

1. We have a current 'complex' model. We run it on the training dataset and obtain the class probability distribution at the output.
2. Next, we train a student model (a model with fewer trees in this case) to replicate the outputs of the teacher model, using the class distribution as the target variable.
3. It is important to note that we do not use the dataset markup in any way, and therefore we can utilize arbitrary data. Naturally, we use a sample of live stream data as a training set for the student model. This way, the training set allows us to ensure the model's accuracy, while the stream sample guarantees similar performance in production distribution, compensating for the bias of the training set.

The combination of these two techniques (adding trees and periodically reducing their number through Knowledge Distillation) introduces new patterns and ensures complete continuity.

Using KD, we also differentiate operations with model features, such as removing features and dealing with missing values. In our case, we have a number of important statistical features (based on senders, text hashes, URLs, etc.) that are stored in a database with the property of failing. The model is, of course, not prepared for such developments, as failure scenarios do not appear in the training set. In such cases, we combine KD techniques and augmentation: during training for a portion of the data, we remove or nullify necessary features, while the labels (outputs of the current model) come from the original samples, and the student model learns to replicate this distribution.

We have noticed that the more serious the manipulation of models, the more stream sample is required in percentage terms.

For the simplest operation of feature removal, only a small part of the stream is needed, as only a couple of features change, and the current model was trained on the same set—the difference is minimal. For model simplification (reducing the number of trees several times), it already requires a 50-50 balance. And for missing important statistical features that significantly affect the model's performance, even more stream is needed to equalize the work of the new model, which is robust to missing values, across all types of emails.

FastText

Let's move on to FastText. Let me remind you that the representation (Embedding) of a word consists of the sum of the embedding of the word itself and all its character N-grams, usually trigrams. Since there can be quite a few trigrams, Bucket Hashing is used, which transforms the entire space into a fixed hash map. As a result, the weight matrix ends up being the internal layer's dimensionality multiplied by the number of words plus the buckets.

In fine-tuning, new features arise: words and trigrams. In the standard fine-tuning from Facebook, nothing significant happens. Only old weights are fine-tuned with cross-entropy on new data. Thus, new features are not utilized; of course, this approach has all the previously described drawbacks related to the unpredictability of the model in production. Therefore, we slightly enhanced FastText. We add all new weights (words and trigrams), retrain the entire matrix with cross-entropy, and introduce harmonic regularization similar to the linear model, which ensures minimal changes to the old weights.

CNN

With convolutional networks, the situation is a bit more complex. If in CNN the last layers are fine-tuned, then, of course, we can apply harmonic regularization and guarantee continuity. However, if the entire network needs to be fine-tuned, such regularization cannot be applied to all layers. Nevertheless, there is an option to train complementary embeddings using Triplet Loss (original article).

Triplet Loss

Using the example of the anti-phishing task, let's discuss Triplet Loss in general terms. We take our logo, as well as positive and negative examples of logos from other companies. We minimize the distance between the first set and maximize the distance between the second set, allowing for a small margin to ensure greater compactness of classes.

If we fine-tune the network, the metric space completely changes, becoming absolutely incompatible with the previous one. This poses a serious problem in tasks that utilize vectors. To circumvent this issue, we will incorporate old embeddings during training.

We have added new data to the training set and are training the second version of the model from scratch. In the second phase, we finetune our network: first, the last layer is adjusted, and then the entire network is unfrozen. In the process of creating triplets, we calculate only part of the embeddings using the new model, while the rest are computed with the old one. This way, during the finetuning process, we ensure compatibility between the metric spaces of v1 and v2. This is a kind of harmonic regularization.

Architecture as a Whole

If we look at the entire system as an example of anti-spam, the models are not isolated but nested within each other. We take images, text, and other features, and use CNN and Fast Text to get embeddings. Next, classifiers are applied on top of the embeddings to produce scores for different classes (types of emails, spam, presence of a logo). The scores and features then enter a tree forest to make the final decision. Individual classifiers in this scheme allow for a better interpretation of the system's results and a more targeted finetuning of components in case problems arise, rather than presenting all data to the decision trees in a raw form.

As a result, we guarantee continuity at each level. At the lower level in CNN and Fast Text, we use harmonic regularization, while for the classifiers in the middle, we also apply harmonic regularization and score calibration for probability distribution compatibility. Boosting trees, on the other hand, are trained incrementally or through Knowledge Distillation.

Overall, supporting such a nested machine learning system is usually challenging, as any component at the lower level leads to an update of the entire system above it. However, since in our setup each component changes slightly and is compatible with the previous one, the entire system can be updated in bits without the need to retrain the whole structure, allowing for maintenance without significant overhead.

Deploy

We have covered data collection and the finetuning of various types of models, so we move on to deploying them in the production environment.

A/B testing

As I mentioned earlier, during the data collection process, we typically obtain a biased sample that makes it impossible to evaluate the production performance of the model. Therefore, when deploying a model, it is essential to compare it with the previous version to understand how things are actually going, meaning we conduct A/B tests. In reality, the rollout process and analysis of graphs is fairly routine and lends itself well to automation. We gradually roll out our models to 5%, 30%, 50%, and then 100% of users, gathering all available metrics based on the model's responses and user feedback. In case of any significant anomalies, we automatically roll back the model, and for other cases, after accumulating a sufficient number of user clicks, we decide to increase the percentage. Ultimately, we bring the new model to 50% of users completely automatically, and a human approves the rollout to the entire audience, though even this step can be automated.

However, the A/B testing process presents an opportunity for optimization. The fact is that any A/B test is quite lengthy (in our case, it takes from 6 to 24 hours depending on the amount of feedback), making it fairly expensive and resource-limited. In addition, a sufficiently high flow percentage is required for the test to essentially speed up the overall A/B test time (gathering a statistically significant sample to evaluate metrics at a small percentage can take a very long time), which makes the number of A/B slots extremely limited. It is clear that we need to bring only the most promising models into the test, and we receive quite a few of them during the fine-tuning process.

To address this challenge, we trained a separate classifier to predict the success of an A/B test. To do this, we use decision-making statistics, Precision, Recall, and other metrics from the training set, the validation set, and a sample from the flow as features. We also compare the model with the current production one, with heuristics, and take into account the model's complexity. Using all these features, the classifier trained on the testing history scores candidate models, in our case, decision trees, and makes a decision on which one to put into the A/B test.

At the time of implementation, this approach significantly increased the number of successful A/B tests.

Testing & monitoring

Testing and monitoring, surprisingly enough, do not harm our health; rather, they improve it and alleviate stress. Testing helps prevent failures, while monitoring allows for timely detection to minimize user impact.

It's important to understand that sooner or later your system will always make mistakes—this is related to the development cycle of any software. At the beginning of the system's development, there are many bugs until everything stabilizes and the main phase of innovation concludes. However, over time, entropy sets in, and errors re-emerge due to component degradation and changes in data, as I mentioned initially.

Here, I would like to emphasize that any machine learning system must be evaluated in terms of its profitability throughout its entire life cycle. Below is a graph illustrating the operation of a system designed to catch a rare type of spam (the line on the graph is around zero). At one point, due to an incorrectly cached feature, it went haywire. Unfortunately, there was no monitoring for abnormal triggers, resulting in the system saving emails to the 'spam' folder at the decision-making boundary in large volumes. Despite correcting the consequences, the system made so many mistakes that it would not recoup its costs even over five years. This represents a complete failure in terms of the model's life cycle.

Therefore, something as simple as monitoring can become crucial in the life of the model. In addition to standard and obvious metrics, we assess the distribution of responses and model scores, as well as the distribution of key feature values. Using KL divergence, we can compare the current distribution with historical data or A/B test values against the remaining flow, which helps identify anomalies in the model and roll back changes in time.

In most cases, we launch our initial system versions using simple heuristics or models, which we later utilize for monitoring. For example, we monitor the NER model against regular expressions for specific online stores, and if the classifier's coverage decreases compared to them, we investigate the reasons behind it. Another useful application of heuristics!

Summary

Let's revisit the key points of the article.

• Fibdex. We always think about the user: how they'll cope with our mistakes and how they can report them. We must remember that users are not a source of pure feedback for training models, and it needs to be refined with the help of auxiliary ML systems. If there is no way to gather user signal, we look for alternative feedback sources, such as related systems.
• Fine-tuning. The key here is continuity, so we rely on the current production model. We train new models in a way that they don't differ significantly from the previous one by using harmonic regularization and similar techniques.
• Deploy. Auto-deployment based on metrics significantly reduces the time spent on implementing models. Monitoring statistics and decision distribution, as well as the number of false inputs from users, is essential for your peace of mind and productive weekends.

Well, I hope what you've read will help you improve your ML systems more quickly, speed up their market launch, and make them more reliable, reducing the stress of working on them.

Source: habr.com