General overview of the service architecture for evaluating appearance based on neural networks

Introduction

Hello!

In this article, I will share my experience in building a microservices architecture for a project that uses neural networks.

We will discuss the architecture requirements, take a look at various structural diagrams, break down each component of the completed architecture, and evaluate the technical metrics of the solution.

Happy reading!

A few words about the task and its solution

The main idea is to assess a person's attractiveness based on a photo using a ten-point scale.

In this article, we will depart from discussing the neural networks used and the process of data preparation and training. However, in one of the upcoming publications, we will certainly return to a deeper analysis of the evaluation pipeline.

For now, we will outline the evaluation pipeline at a high level, focusing on the interaction of microservices within the overall architecture of the project. 

In working on the attractiveness evaluation pipeline, the task was decomposed into the following components:

1. Face detection in photos
2. Evaluation of each detected face
3. Rendering the result

The first is handled by a pre-trained MTCNN. For the second task, a convolutional neural network was trained on PyTorch, with ResNet34 being used as the backbone to balance 'quality / inference speed on CPU.'

Functional diagram of the evaluation pipeline

Analysis of the architecture requirements for the project

In the lifecycle ML of a project, the stages of working on the architecture and automating the deployment of the model are often among the most time-consuming and resource-intensive.

ML project lifecycle

This project is no exception—we decided to wrap the evaluation pipeline in an online service, which required delving into the architecture. The following basic requirements were outlined:

1. A unified log storage— all services must write logs to a single location, which should be easy to analyze.
2. The ability to horizontally scale the evaluation service—as the most likely bottleneck
3. Each image evaluation should allocate the same amount of CPU resources—to avoid spikes in inference time distribution.
4. Fast (re)deployment of both specific services and the stack as a whole.
5. The ability to use shared objects in different services if necessary.

Architecture

After analyzing the requirements, it became clear that microservice architecture fits almost perfectly.

To avoid unnecessary headaches, Telegram API was chosen as the frontend.

First, let's look at the structural diagram of the completed architecture, then we'll move on to describing each component and formalizing the process of successful image processing.

Structural diagram of the completed architecture

Let's talk in more detail about each component of the diagram, clarifying their Single Responsibility in the image evaluation process.

Microservice 'attrai-telegram-bot'

This microservice encapsulates all interactions with the Telegram API. There are two main scenarios to highlight – working with the user’s image and handling the results of the evaluation pipeline. We'll discuss both scenarios in general.

Upon receiving a user message with an image:

1. Filtering is performed, consisting of the following checks:
   • The optimal size of the image
   • The number of user images already in the queue
2. If the primary filtering is passed, the image is saved in a docker volume
3. A task is produced to the 'to_estimate' queue, which includes the path to the image stored in our volume
4. If the above steps are successfully completed, the user will receive a message with an estimated processing time for the image, calculated based on the number of tasks in the queue. In case of an error, the user will be explicitly notified by a message with information on what might have gone wrong.

Additionally, this microservice, as a celery worker, listens to the 'after_estimate' queue, which is intended for tasks that have passed through the evaluation pipeline.

Upon receiving a new task from 'after_estimate':

1. If the image was processed successfully, we send the result to the user; if not, we notify them of the error.
2. We delete the image that is the result of the evaluation pipeline.

Evaluation microservice 'attrai-estimator'

This microservice is a celery worker and encapsulates everything related to the image evaluation pipeline. The workflow here is straightforward – let's break it down.

Upon receiving a new task from 'to_estimate':

1. We run the image through the evaluation pipeline:
   1. We load the image into memory.
   2. Resizing the image to the required dimensions
   3. Detecting all faces (MTCNN)
   4. Evaluating all faces (wrapping the faces found in the previous step into a batch and inferring with ResNet34)
   5. Rendering the final image
      1. Drawing bounding boxes
      2. Drawing evaluations
2. Removing the user (original) image
3. Saving the output from the evaluation pipeline
4. Placing the task in the queue 'after_estimate', which is listened to by the previously discussed microservice 'attrai-telegram-bot'

Graylog (+ mongoDB + Elasticsearch)

Graylog — is a solution for centralized log management. In this project, it was used for its intended purpose.

The choice was made specifically for it, rather than the familiar ELK stack, due to the convenience of working with it from Python. Everything needed for logging in Graylog is to add GELFTCPHandler from the graypy package to the other root logger handlers of our Python microservice.

As someone who previously worked only with the ELK stack, I generally had a positive experience while working with Graylog. The only downside is the superiority of Kibana's features over Graylog's web interface.

RabbitMQ

RabbitMQ — is a message broker based on the AMQP protocol.

In this project, it was used as the most stable and time-tested broker for Celery and operated in durable mode.

Redis

Redis — is a NoSQL DBMS working with data structures of the 'key-value' type.

Sometimes there is a need to use common objects across different Python microservices that implement certain data structures.

For example, Redis stores a hashmap of 'telegram_user_id => number of active tasks in the queue', which allows limiting the number of requests from a single user to a specific value and thus prevent DoS attacks.

Formalizing the process of successfully processing the image

1. The user sends an image to the Telegram bot
2. ‘attrai-telegram-bot’ receives the message from the Telegram API and parses it
3. The task with the image is added to the asynchronous queue ‘to_estimate’
4. The user receives a message with the expected evaluation time
5. ‘attrai-estimator’ takes the task from the ‘to_estimate’ queue, processes it through the evaluation pipeline, and produces a task in the ‘after_estimate’ queue
6. ‘attrai-telegram-bot’, listening to the ‘after_estimate’ queue, sends the result to the user

DevOps

Finally, after reviewing the architecture, we can move on to the no less interesting part — DevOps

the development process and in the production environment? Or use the same file for

the development process and in the production environment? Or use the same file for  — a clustering system that is implemented within Docker Engine and is available out of the box.

With the help of the 'swarm', all nodes of our cluster can be divided into two types – worker and manager. Containers (stacks) are deployed on machines of the first type, while machines of the second type are responsible for scaling, balancing, and other cool features.By default, managers also act as workers.

A cluster with one leader manager and three workers.

The minimal possible size of a cluster is one node; a single machine will act as both a leader manager and a worker. Based on the project size and minimal requirements for fault tolerance, this approach was chosen.

To get ahead of the story, I will say that since the first production delivery, which was in mid-June, there have been no issues related to this clustering organization (but that doesn't mean that such organization is acceptable for any medium-sized projects, which have fault tolerance requirements).

Docker Stack

In 'swarm' mode, the deployment of stacks (sets of Docker services) is handled by docker stack.

It supports docker-compose configs, allowing additional use of deploy parameters.  

For example, with the help of these parameters, resources were limited for each instance of the evaluation microservice (allocating N cores for N instances, and in the microservice itself limiting the number of cores used by PyTorch to one).

attrai_estimator:
  image: 'erqups/attrai_estimator:1.2'
  deploy:
    replicas: 4
    resources:
      limits:
        cpus: '4'
    restart_policy:
      condition: on-failure
      …

It is important to note that Redis, RabbitMQ, and Graylog are stateful services and scaling them as easily as 'attrai-estimator' is not possible.

Anticipating the question — why not Kubernetes?

It seems that using Kubernetes for small and medium-sized projects is overkill; all necessary functionality can be obtained from Docker Swarm, which is quite user-friendly for container orchestration and has a low entry barrier.

Infrastructure

This was deployed on a VDS with the following specifications:

• CPU: 4 cores Intel® Xeon® Gold 5120 CPU @ 2.20GHz
• RAM: 8 GB
• SSD: 160 GB

After local stress testing, it seemed that this machine would be just barely enough under a significant influx of users.

However, right after the deployment, I posted a link to one of the most popular imageboards in the CIS (yes, that one), which sparked interest, and within a few hours, the service successfully processed tens of thousands of images. At peak moments, CPU and RAM resources were not even half utilized.

A bit more graphics

The number of unique users and requests for evaluation since the deployment, depending on the day

Distribution of the inference time of the evaluation pipeline

Conclusions

In summary, I can say that the architecture and approach to container orchestration have fully justified themselves — even at peak moments, there were no crashes or slowdowns in processing time. 

I believe that small and medium-sized projects that utilize real-time inference of neural networks on CPUs can successfully adopt the practices described in this article.

I should add that the article was originally longer, but to avoid posting a long read, I decided to omit certain points — we will return to them in future publications.

You can poke at the bot on Telegram — @AttraiBot, it will work, at least until the end of autumn 2020. I remind you — no user data is stored — neither the original images nor the results of the evaluation pipeline — everything is deleted after processing.

Source: habr.com