Every service whose users can create their own content (UGC - User-generated content) has to solve business tasks as well as maintain order in UGC. Poor or inadequate content moderation can ultimately reduce the service's attractiveness to users, even to the point of ceasing its operation.
Today we will tell you about the synergy between Yula and Odnoklassniki, which helps us effectively moderate ads on Yula.
Synergy is generally very useful, and in the modern world, where technologies and trends change rapidly, it can become a crucial resource. Why spend limited resources and time inventing what has already been created and perfected?
We also thought this way when we faced the task of moderating user-generated content — images, text, and links. Our users upload millions of content items to Yula every day, and without automated processing, it is simply impossible to manually moderate all this data.
Therefore, we leveraged an existing moderation platform that our colleagues from Odnoklassniki had refined to a state of 'almost perfection' by that time.
Why Odnoklassniki?
Every day, tens of millions of users come to the social network, publishing billions of content items: from photos to videos and texts. The Odnoklassniki moderation platform helps verify very large volumes of data and combat spammers and bots.
The moderation team at OК has accumulated a wealth of experience, as they have been enhancing their tool for 12 years. Importantly, they could not only share their ready-made solutions but also adapt the architecture of their platform to our specific needs.
From now on, for brevity, we will simply refer to the OК moderation platform as 'the platform'.
How Everything Works
Data exchange between Yula and Odnoklassniki is established through .
Why we chose this tool:
- In Yula, all ads undergo post-moderation, so initially, a synchronous response was not required.
- If a fierce outage occurs and Yula or Odnoklassniki become unavailable, including due to peak loads, then the data from Kafka will not be lost and it can be read later.
- The platform has already been integrated with Kafka, so most security issues have been resolved.
For every user-created or modified ad on Yula, a JSON file with the data is generated and placed into Kafka for subsequent moderation. Ads are loaded from Kafka into the platform, where decisions are made either automatically or manually. Poor ads are blocked with a reason provided, while those that the platform deems compliant are marked as 'good'. All decisions are then sent back to Yula and applied in the service.
Ultimately, for Yula, everything boils down to simple actions: send an ad to the Odnoklassniki platform and receive a resolution of 'ok', or an explanation of why it is not 'ok'.
Automatic processing
What happens to the ad after it reaches the platform? Each ad is divided into several entities:
- title,
- description,
- photos,
- the user-selected category and subcategory of the ad,
- price.
Then, the platform clusters each entity to find duplicates. The text and photos are clustered using different schemes.
The texts are normalized before clustering to eliminate special characters, modified letters, and other noise. The resulting data is broken down into N-grams, each of which is hashed. In the end, a multitude of unique hashes is generated. The similarity between texts is determined by between the two resulting sets. If the similarity exceeds the threshold, the texts are merged into one cluster. To speed up the search for similar clusters, MinHash and Locality-sensitive hashing are used.
Various methods for merging images have been devised, from comparing pHash of pictures to finding duplicates with the help of neural networks.
The last method is the most "severe." For training the model, such image triples (N, A, P) were selected where N is unlike A, and P is similar to A (a partial duplicate). The neural network then learned to make A and P as similar as possible, while maximizing the distance between A and N. This results in fewer false positives compared to simply taking embeddings from a pre-trained network.
When the neural network receives images as input, it generates an N(128)-dimensional vector for each of them and makes a request to assess the similarity of the images. A threshold is then calculated, beyond which similar images are considered duplicates.
The model is skilled at finding spammers who purposely photograph the same item from different angles to circumvent pHash comparisons.
An example of spam photos, stitched together by the neural network as duplicates.
At the final stage, duplicate listings are searched based on both text and images simultaneously.
If two or more listings cluster together, the system initiates automatic blocking, which uses specific algorithms to determine which duplicates to delete and which to keep. For instance, if two users have identical photos in their listings, the system will block the newer listing.
After creation, all clusters go through a series of automatic filters. Each filter assigns a score to the cluster: the likelihood that it contains a threat identified by that filter.
For example, the system analyzes the description in the listing and selects potential categories for it. It then takes the one with the highest probability and compares it with the category specified by the listing author. If they do not match, the listing is blocked for incorrect categorization. And since we are kind and honest, we directly inform the user which category they need to choose for the listing to pass moderation.
Notification of blocking due to incorrect category.
On our platform, machine learning feels at home. For instance, we use it to search for prohibited items in titles and descriptions within Russia. The neural network models carefully "examine" images for any URLs, spam texts, phone numbers, and the forbidden content.
In cases where banned goods are attempted to be sold disguised as something legal, and there is no text in either the title or description, we use image tagging. Each image can have up to 11 thousand different tags that describe what is depicted in the image.
A hookah is attempted to be sold disguised as a samovar.
Alongside complex filters, simple ones are also at work, addressing obvious text-related tasks:
- profanity;
- URL and phone number detectors;
- mentions of messengers and other contacts;
- unreasonably low prices;
- ads where nothing is being sold, etc.
Today, every ad passes through a fine sieve of over 50 automated filters that try to detect something inappropriate in the ad.
If none of the detectors are triggered, a response is sent to Yulu indicating that the ad is 'most likely' in complete order. We apply this response on our platform, and users who subscribed to the seller receive a notification about a new product.
Notification that the seller has a new product.
As a result, each ad 'accumulates' metadata, some of which is generated when creating the ad (author's IP address, user-agent, platform, geolocation, etc.), while the rest consists of scores issued by each filter.
Ad queues
When an ad enters the platform, the system places it in one of the queues. Each queue is formed using a mathematical formula that combines the ad's metadata in a way that aims to detect any bad patterns.
For instance, an ad queue in the 'Mobile Phones' category can be created from Yulu users supposedly located in Saint Petersburg, while their IP addresses are from Moscow or other cities.
Examples of ads posted by one user in different cities.
Or queues can be formed based on scores assigned to ads by a neural network, arranged in descending order.
Each queue, according to its formula, assigns a final score to the ad. Thereafter, different actions can be taken:
- set a threshold value at which the ad will receive a specific type of blocking;
- send all listings in the queue to moderators for manual review;
- or combine the previous options: set a threshold for automatic blocking and send to moderators those listings that did not reach this threshold.
Why are these queues needed? Suppose a user uploads a photo of a firearm. The neural network assigns it a score from 95 to 100 and determines with 99% accuracy that there is a weapon in the picture. But if the score is below 95%, the model's accuracy begins to decline (this is a feature of neural network models).
As a result, a queue is formed based on the model's score, and listings that received a score from 95 to 100 are automatically blocked as 'Prohibited items.' Listings with a score below 95 are sent for manual review by moderators.
Chocolate Beretta with cartridges. For manual moderation only! 🙂
Manual moderation
As of early 2019, about 94% of all listings on Yula are moderated automatically.
If the platform cannot decide on certain listings, it sends them for manual moderation. Odnoklassniki developed their own tool: in the tasks for moderators, all the necessary information for making a quick decision is immediately displayed – whether the listing is valid or should be blocked with a stated reason.
To ensure that the quality of service is not compromised during manual moderation, the work of people is constantly monitored. For example, in the stream of tasks, the moderator is shown 'traps' – listings for which there are already ready-made solutions. If the moderator's decision does not match the ready-made one, the moderator is marked as having made an error.
On average, a moderator spends 10 seconds reviewing one listing. The error rate is no more than 0.5% of all reviewed listings.
Community moderation
Colleagues from Odnoklassniki went even further, utilizing 'hall assistance': they wrote a game application for the social network, allowing rapid labeling of large amounts of data by highlighting some unwanted characteristic – Moderator of Odnoklassniki (). A good way to engage O.K. users to help make content more enjoyable.
A game where users mark photos that contain a phone number.
Any queue of advertisements in the platform can be redirected to the game Moderator of Odnoklassniki. Everything that users of the game mark is then sent to internal moderators for review. This scheme allows blocking ads for which filters have not yet been created, while simultaneously creating training samples.
Storage of moderation results
We save all decisions made during moderation to avoid reprocessing the advertisements that have already been reviewed.
Daily, millions of clusters are created for the advertisements. Over time, each cluster receives a 'good' or 'bad' mark. Each new advertisement or its revision that falls into a marked cluster automatically receives the cluster's resolution. There are about 20,000 such automatic resolutions each day.
If no new advertisements are received in a cluster, it is deleted from memory, and its hash and decision are recorded in Apache Cassandra.
When the platform receives a new advertisement, it first tries to find a similar cluster among the already created ones and take the decision from there. If no such cluster exists, the platform goes to Cassandra and searches there. Found it? Great, it applies the decision to the cluster and sends it to Yula. An average of about 70,000 'repeat' decisions accumulate daily — accounting for 8% of the total.
In summary
We have been using the moderation platform of Odnoklassniki for two and a half years. We like the results:
- 94% of all advertisements are automatically moderated within a day.
- The cost of moderating one advertisement has decreased from 2 rubles to 7 kopecks.
- Thanks to the ready-made tool, we forgot about the problems of managing moderators.
- We increased the number of manually processed advertisements by 2.5 times while maintaining the same number of moderators and budget. Additionally, the quality of manual moderation has improved due to automated control, hovering around 0.5% errors.
- We quickly cover new types of spam with filters.
- We swiftly connect new divisions to moderation. . Since 2017, new verticals for Real Estate, Job Vacancies, and Auto have appeared in Yula.
Source: habr.com
