When your job is to create something beautiful, there's not much to say about it because the result is visible to everyone. However, if you're erasing graffiti from fences, no one notices your work until the fences look decent or until you erase something inappropriate.
Any service that allows comments, reviews, messages, or image uploads will eventually face issues with spam, fraud, and inappropriate content. This is unavoidable, but it must be addressed.
My name is Mikhail, and I work in the Anti-Spam team, which protects users of Yandex services from such issues. Our work is rarely noticeable (and that's good!), so today I will tell you more about it. You will learn when moderation is ineffective and why accuracy is not the only measure of its effectiveness. We will also discuss profanity using the example of cats and dogs and why sometimes it's useful to 'think like a profanity user.'
Yandex is launching more and more services where users publish their content. You can ask questions or write answers in Yandex.Q, discuss local news in Yandex.District, and share traffic updates in Yandex.Maps chats. However, as the audience of a service grows, it becomes attractive to scammers and spammers. They come and clutter the comments: suggesting easy money, advertising miraculous remedies, and promising social benefits. Because of spammers, some users lose money while others lose the desire to spend time on a neglected service filled with spam.
And that's not the only problem. We strive not only to protect users from scammers but also to create a comfortable atmosphere for communication. If people encounter profanity and insults in comments, they are highly likely to leave and not return. Thus, we need to tackle this issue as well.
Clean Web
As is often the case with us, the first developments arose in Search, particularly in the segment dedicated to combating spam in the results. About ten years ago, the task of filtering adult content for family-friendly searches and queries that do not warrant responses from the 18+ category emerged. This led to the creation of the first manually compiled dictionaries of adult content and profanity, which were supplemented by analysts. The main objective was to classify queries into those where adult content could be shown and those where it could not. For this task, we collected annotations, built heuristics, and trained models, leading to the initial groundwork for filtering undesirable content.
Over time, Yandex began to feature UGC (user-generated content) — messages authored by users themselves, with Yandex merely publishing them. For the reasons described above, many messages could not be published blindly — moderation was required. Accordingly, it was decided to create a service that would provide protection against spam and malicious actors for all Yandex UGC products and utilize existing frameworks for filtering undesirable content in Search. The service was named 'Clean Web.'
New tasks and support from crowdworkers
Initially, we only had simple automation: services would send us texts, and we would run profanity lists, adult content dictionaries, and regular expressions on them — analysts compiled everything manually. However, over time, the service was applied to an increasing number of Yandex products, and we had to learn to deal with new challenges.
Often, instead of reviews, users post a nonsensical string of letters in an attempt to inflate their achievements; sometimes they advertise their company in reviews of a competitor, and sometimes they simply confuse organizations, posting comments like, 'Excellently cooked fish!' in a review for a pet store. Perhaps one day, artificial intelligence will learn to perfectly grasp the meaning of any text, but for now, automation sometimes performs worse than a human.
It became clear that manual markup was unavoidable, so we added a second stage to our workflow — submission for manual review by a person. Texts that the classifier did not flag as problematic were sent there. You can easily imagine the scale of this task, so we not only relied on assessors but also leveraged the 'wisdom of the crowd' by reaching out to crowdworkers. They help us identify what the machine missed, thereby training it.
Smart caching and LSH hashing
Another problem we encountered while working with comments was spam, specifically its volume and rate of spread. As the audience of Yandex.Zone grew rapidly, spammers arrived. They learned to bypass regex patterns by slightly altering the text. Spam messages were indeed found and removed, but on the scale of Yandex, a message deemed unacceptable for even 5 minutes could be seen by hundreds of people.
Naturally, we were not satisfied with this, so we implemented smart text caching based on LSH (). This works as follows: we normalized the text, removed links, and divided it into n-grams (sequences of n letters). Then we calculated hashes from the n-grams and constructed the LSH vector of the document based on them. The idea is that similar texts, even if slightly modified, turned into similar vectors.
This solution allowed us to reuse the verdicts of classifiers and crowdworkers for similar texts. During a spam attack, as soon as the first message passed review and entered the cache with a 'spam' verdict, all new similar messages, even if modified, received the same verdict and were automatically deleted. Later, we learned to train and retrain spam classifiers automatically, but this 'smart cache' has stayed with us and often comes to our rescue.
Classifier of good texts
Before we could catch our breath from battling spam, we realized that 95% of our content is moderated manually: classifiers only respond to violations, while most texts are good. We rely on raters, who in 95 cases out of 100 rate content as 'All OK'. We had to embark on an unfamiliar task — creating classifiers for good content, fortunately, we had accumulated enough annotations over time.
The first classifier looked like this: we lemmatize the text (bringing words to their base form), discard all functional parts of speech, and apply a pre-prepared 'dictionary of good lemmas'. If all words in the text are 'good', then the text as a whole does not contain violations. This approach immediately provided 25 to 35% automation of manual markup across various services. Of course, this method is not perfect: it is easy to combine several innocent words and produce a very offensive statement, but it allowed us to quickly achieve a good level of automation and gave us time to train more complex models.
Subsequent versions of the good text classifiers already included linear models, decision trees, and their combinations. For marking rudeness and insults, we are trying the BERT neural network, for example. Here, it is important to capture the meaning of a word in context and the relationship between words across different sentences, and BERT handles this well. (By the way, recently, colleagues from the News department demonstrated how they use the technology for an unconventional task — finding errors in headlines.) As a result, we managed to automate up to 90% of the flow depending on the service. Accuracy, completeness, and speed
To evolve, we need to understand the benefits provided by various automatic classifiers, changes in them, and whether the quality of manual checks is degrading. For this, we apply accuracy and completeness metrics.
To grow, it's essential to understand the benefits provided by various automatic classifiers, changes made to them, and to ensure that the quality of manual checks does not deteriorate. For this, we use accuracy and completeness metrics.
Accuracy is the proportion of correct judgments among all judgments about poor content. The higher the accuracy, the fewer false positives. If accuracy is not monitored, in theory, you could remove all spam and profanity, together with half of the good messages. On the other hand, relying solely on accuracy means that the best technology would be one that catches no one at all. Therefore, there is also a measure of completeness: the proportion of detected bad content among the total volume of bad content. These two metrics balance each other.
To measure, we sample the entire incoming flow for each service and provide content samples to assessors for expert evaluation and comparison with machine decisions.
But there is one more important metric.
I mentioned earlier that an unacceptable message can be seen by hundreds of people even within 5 minutes. Therefore, we count how many times we managed to show people poor content before we hid it. This is important because it's not enough to work effectively — you also need to work quickly. And when we were building defenses against profanity, we felt this acutely.
Anti-profane Measures Using Cats and Dogs as Examples
A brief lyrical digression. Some might argue that obscenity and insults are not as dangerous as malicious links, and not as irritating as spam. However, we aim to maintain a comfortable environment for millions of users, as people do not like to return to places where they are insulted. It's no coincidence that bans on profanity and insults are outlined in the rules of many communities, including on Habr. But I digress.
Profanity dictionaries do not cope with the richness of the Russian language. Although there are only four main roots of obscene words, an immeasurable number of words can be formed from them, which cannot be caught by any regular expressions. Additionally, one can write part of a word in transliteration, replace letters with similar combinations, rearrange letters, add asterisks, and so on. Sometimes, it is virtually impossible to determine what the user meant without context. We respect Habr's rules, so we will demonstrate this not with real examples, but with cats and dogs.
"Mew," said the kitten. But we understand that the kitten meant another word...
We started to think about the algorithms for 'fuzzy matching' in our dictionary and smarter preprocessing: converting transliteration, removing spaces and punctuation, looking for patterns, and writing separate regular expressions for them. This approach yielded results, but often reduced accuracy, failing to provide the desired completeness.
Then we decided to 'think like the profane'. We began to introduce noise into the data ourselves: rearranging letters, generating typos, substituting letters with similar-looking ones, and so on. We took the initial markup for this by applying profanity dictionaries to large text corpuses. If you take one sentence and distort it in several ways, you will end up with many different sentences. This can significantly increase the training sample size. All that was left was to train some reasonably intelligent model on the resulting pool, taking context into account.
It's still too early to talk about a final solution. We are still experimenting with approaches to this problem, but we can already see that a simple character-based convolutional network with several layers significantly outperforms dictionaries and regular expressions: it increases both accuracy and completeness.
Of course, we understand that there will always be ways to bypass even the most advanced automation, especially when it comes to such a playful endeavor: writing in a way that a dumb machine won't understand. Here, just like in the fight against spam, our goal is not to eradicate the possibility of writing something obscene; our task is to ensure that the game isn't worth the candle.
Opening up the opportunity to share opinions, communicate, and comment is not difficult. It is much harder to achieve safe, comfortable conditions and respect for individuals. Without this, there will be no development of any community.
Source: habr.com
