Hello everyone, friends!
* This article is inspired by the open workshop REBRAIN & Yandex.Cloud; if you prefer to watch a video, you can find it at this link —
Recently, we had the chance to experience Yandex.Cloud firsthand. Since we wanted to immerse ourselves deeply, we immediately abandoned the idea of launching a simple WordPress blog with a cloud database — too boring. After some brief deliberation, we decided to deploy something resembling a production architecture for accepting and analyzing events in near real-time.
I am absolutely sure that the overwhelming majority of online (and not only) businesses collect a mountain of information about their users and their actions in one way or another. At the very least, this is necessary for making certain decisions — for example, if you manage an online game, you might want to look at the statistics of which levels users most often get stuck on and delete your game. Or why users leave your site without making a purchase (hello, Yandex.Metrica).
So, our story: how we wrote an application in Go, tested Kafka vs RabbitMQ vs Yandex Queue Service, developed data streaming to the ClickHouse cluster, and visualized data using Yandex Datalens. Naturally, all of this was seasoned with infrastructure delights like Docker, Terraform, GitLab CI, and, of course, Prometheus. Let’s go!
First, I want to clarify that we won't be able to set everything up in one go — we’ll need several articles in the series for that. A little about the structure:
Part 1 (which you are reading). We will define the requirements and architecture of the solution, as well as write the application in Go.
Part 2. We will deploy our application to production, make it scalable, and test the load.
Part 3. We will try to figure out why we need to store messages in a buffer rather than in files, and compare Kafka, RabbitMQ, and Yandex Queue Service.
Part 4. We will be deploying a ClickHouse cluster, writing a stream to transfer data from the buffer there, and setting up visualization in Datalens.
Part 5. We will bring all infrastructure into proper shape — we will set up CI/CD using GitLab CI, connect monitoring and service discovery with Prometheus and Consul.
Technical Specification
First, we will formulate the technical assignment — what exactly we want to achieve as a result.
- We want to have an endpoint like events.kis.im (kis.im is a test domain that we will use throughout all articles) that should receive events via HTTPS.
- Events are a simple JSON object like: {"event": "view", "os": "linux", "browser": "chrome"}. In the final stage, we will add a few more fields, but it won't make a significant difference. If desired, we can switch to protobuf.
- The service must be able to handle 10,000 events per second.
- There should be a possibility to scale horizontally by simply adding new instances to our solution. It would also be good if we could deploy the frontend in various geolocations to reduce latency for client requests.
- Fault tolerance. The solution must be stable enough to survive the failure of any components (up to a certain number, of course).
Architecture
For such tasks, classic architectures have long been devised that allow for efficient scaling. The diagram shows an example of our solution.
So, what do we have:
1. On the left, we have our devices that generate various events, whether it's players progressing through levels in a game on their smartphones or placing orders in an online store via a regular browser. An event, as specified in the technical specification, is a simple JSON object sent to our endpoint — events.kis.im.
2. The first two servers are simple load balancers, and their main tasks are:
- To be constantly available. For this, we can use, for example, keepalived, which will switch the virtual IP between nodes in case of issues.
- To terminate TLS. Yes, we will terminate TLS on them. First, to comply with the technical specification, and second, to offload the burden of establishing encrypted connections from our backend servers.
- To balance incoming requests among available backend servers. The key word here is available. Based on this, we come to the understanding that load balancers must be able to monitor our servers with applications and cease to balance traffic to failed nodes.
3. Behind the load balancers, we have application servers running a relatively simple application. It must be able to accept incoming HTTP requests, validate the received JSON, and store data in a buffer.
4. In the diagram, Kafka is depicted as a buffer, though, of course, other similar services can be utilized at this level. We will compare Kafka, RabbitMQ, and YQS in the third article.
5. The penultimate point of our architecture is ClickHouse — a columnar database that allows for the storage and processing of massive amounts of data. At this level, we need to transfer data from the buffer to the actual storage system (we will discuss this in the fourth article).
This scheme enables us to independently scale each layer horizontally. If the backend servers cannot handle the load, we can simply add more, as they are stateless applications, allowing for automated scaling. If the Kafka buffer cannot keep up, we will add more servers and migrate some partitions of our topic onto them. If ClickHouse cannot handle the load — well, that's impossible 🙂 In reality, we will also add servers and shard the data.
By the way, if you want to implement the optional part of our specifications and enable scaling in various geolocations, it couldn't be easier:
In each geolocation, we deploy a load balancer with applications and Kafka. In general, two application servers, three Kafka nodes, and a cloud balancer, such as Cloudflare, which will check the availability of application nodes and balance requests by geolocation based on the client's originating IP address, will suffice. This way, data sent by an American client will land on American servers, while data from Africa will go to African servers.
Next, everything is quite straightforward — we use the mirror tool from the Kafka toolkit and copy all data from all locations to our central data center located in Russia. Inside, we parse the data and save it to ClickHouse for further visualization.
So, we've figured out the architecture — let's start swinging Yandex.Cloud!
Writing an Application
We still have to wait a bit for Cloud and write a fairly simple service for processing incoming events. We will use Golang because it has proven to be a very effective language for writing network applications.
After spending an hour (maybe a couple of hours), we obtain something like this: .
What key points would you like to highlight here:
1. When starting the application, two flags can be specified. One controls the port on which we will listen for incoming HTTP requests (-addr). The other is for the address of the Kafka server where we will publish our events (-kafka):
addr = flag.String("addr", ":8080", "TCP address to listen to")
kafka = flag.String("kafka", "127.0.0.1:9092", "Kafka endpoints")2. The application uses the Sarama library () to send messages to the Kafka cluster. We immediately set configurations aimed at maximizing processing speed:
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForLocal
config.Producer.Compression = sarama.CompressionSnappy
config.Producer.Return.Successes = true3. Our application also integrates the Prometheus client, which collects various metrics, such as:
- the number of requests to our application;
- the number of errors when processing requests (unable to read POST requests, broken JSON, unable to write to Kafka);
- the time taken to process a single request from the client, including the time to write the message to Kafka.
4. Three endpoints that our application handles:
- /status — просто возвращаем ok, чтобы показать, что мы живы. Хотя можно и добавить некоторые проверки, типа доступности кафка кластера.
- /metrics — по этому url prometheus client будет возвращать собранные им метрики.
- /post — основной endpoint, куда будут приходить POST запросы с json внутри. Наше приложение проверяет json на валидность и если все ок — записывает данные в кафка-кластер.
I should note that the code is not perfect — it can (and should!) be improved. For instance, you could eliminate the built-in net/http and switch to the faster fasthttp. Or save processing time and CPU resources by postponing JSON validation to a later stage — when data is moved from the buffer to the ClickHouse cluster.
Besides the development aspect, we immediately considered our future infrastructure and decided to deploy our application via Docker. The final Dockerfile for building the application is . Overall, it is quite simple; the only point to emphasize is the multistage build which reduces the final size of our container image.
First Steps in the Cloud
First, we register at . After filling in all the necessary fields, an account will be created for us and we will be given a grant for a certain amount of money to use for testing cloud services. If you wish to repeat all the steps from our article, this grant should be sufficient.
After registration, a separate cloud and a default catalog will be created for you where you can start creating cloud resources. In Yandex.Cloud, the interrelation of resources looks as follows:
One account can be used to create multiple clouds. Inside each cloud, you can create different directories for various company projects. You can read more about this in the documentation — . By the way, I will frequently refer to it later in the text. When I was setting up the entire infrastructure from scratch — the documentation helped me out more than once, so I recommend studying it.
You can manage the cloud using both the web interface and the command-line utility — yc. Installation is done with a single command (for Linux and Mac OS):
curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bashIf your internal security guard is getting agitated about running scripts from the internet — first of all, you can open the script and read it, and secondly, we are running it under our user — without root privileges.
If you want to install the client for Windows, you can follow the instructions and then execute yc init, to set it up completely:
vozerov@mba:~ $ yc init
Welcome! This command will take you through the configuration process.
Please go to https://oauth.yandex.ru/authorize?response_type=token&client_id= in order to obtain your OAuth token.
Please enter your OAuth token:
Please select a cloud to use:
[1] cloud-b1gv67ihgfu3bp (id = b1gv67ihgfu3bpt24o0q)
[2] fevlake-cloud (id = b1g6bvup3toribomnh30)
Please enter your numeric choice: 2
Your current cloud has been set to 'fevlake-cloud' (id = b1g6bvup3toribomnh30).
Please choose a folder to use:
[1] default (id = b1g5r6h11knotfr8vjp7)
[2] Create a new folder
Please enter your numeric choice: 1
Your current folder has been set to 'default' (id = b1g5r6h11knotfr8vjp7).
Do you want to configure a default Compute zone? [Y/n]
Which zone do you want to use as a profile default?
[1] ru-central1-a
[2] ru-central1-b
[3] ru-central1-c
[4] Don't set default zone
Please enter your numeric choice: 1
Your profile default Compute zone has been set to 'ru-central1-a'.
vozerov@mba:~ $In principle, the process isn't complicated — first, you need to obtain an OAuth token to manage the cloud, select the cloud and folder that you will use.
If you have multiple accounts or folders within a single cloud, you can create additional profiles with separate settings using yc config profile create and switch between them.
In addition to the aforementioned methods, the Yandex.Cloud team has created a very good for managing cloud resources. On my side, I prepared a git repository where I described all the resources that will be created within this article — . We're interested in the master branch, so let's clone it locally:
vozerov@mba:~ $ git clone https://github.com/rebrainme/yandex-cloud-events/ events
Cloning into 'events'...
remote: Enumerating objects: 100, done.
remote: Counting objects: 100% (100/100), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 100 (delta 37), reused 89 (delta 26), pack-reused 0
Receiving objects: 100% (100/100), 25.65 KiB | 168.00 KiB/s, done.
Resolving deltas: 100% (37/37), done.
vozerov@mba:~ $ cd events/terraform/All the main variables used in Terraform are specified in the main.tf file. To get started, create a private.auto.tfvars file in the terraform folder with the following content:
# Yandex Cloud Oauth token
yc_token = ""
# Yandex Cloud ID
yc_cloud_id = ""
# Yandex Cloud folder ID
yc_folder_id = ""
# Default Yandex Cloud Region
yc_region = "ru-central1-a"
# Cloudflare email
cf_email = ""
# Cloudflare token
cf_token = ""
# Cloudflare zone id
cf_zone_id = ""All variables can be obtained from yc config list, as we have already set up the command-line utility. I recommend immediately adding private.auto.tfvars to .gitignore to avoid accidentally publishing private data.
In private.auto.tfvars, we also specified the Cloudflare data — for creating DNS records and proxying the main domain events.kis.im to our servers. If you do not wish to use Cloudflare, remove the Cloudflare provider initialization from main.tf and the dns.tf file, which is responsible for creating the necessary DNS records.
In our work, we will combine all three methods — web interface, command-line utility, and Terraform.
Virtual Networks
Honestly, this step could be skipped, as when you create a new cloud, a separate network and three subnets are automatically created — one for each availability zone. However, we would still like to create a separate network for our project with its own addressing. The overall network layout in Yandex.Cloud is shown in the picture below (honestly taken from )
So, you create a shared network within which resources can communicate with each other. For each availability zone, a subnet is created with its own addressing and connected to the shared network. As a result, all cloud resources within it can communicate, even if they are in different availability zones. Resources connected to different cloud networks can see each other only through external addresses. By the way, how this magic works inside .
Network creation is described in the network.tf file from the repository. There, we create one shared private network internal and connect three subnets in different availability zones — internal-a (172.16.1.0/24), internal-b (172.16.2.0/24), internal-c (172.16.3.0/24).
We initialize Terraform and create the networks:
vozerov@mba:~/events/terraform (master) $ terraform init
... skipped ..
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_vpc_subnet.internal-a -target yandex_vpc_subnet.internal-b -target yandex_vpc_subnet.internal-c
... skipped ...
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
yandex_vpc_network.internal: Creating...
yandex_vpc_network.internal: Creation complete after 3s [id=enp2g2rhile7gbqlbrkr]
yandex_vpc_subnet.internal-a: Creating...
yandex_vpc_subnet.internal-b: Creating...
yandex_vpc_subnet.internal-c: Creating...
yandex_vpc_subnet.internal-a: Creation complete after 6s [id=e9b1dad6mgoj2v4funog]
yandex_vpc_subnet.internal-b: Creation complete after 7s [id=e2liv5i4amu52p64ac9p]
yandex_vpc_subnet.internal-c: Still creating... [10s elapsed]
yandex_vpc_subnet.internal-c: Creation complete after 10s [id=b0c2qhsj2vranoc9vhcq]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.Great! We have set up our network and are now ready to create our internal services.
Creating Virtual Machines
For testing the application, we just need to create two virtual machines — the first one will be required for building and running the application, and the second one for running Kafka, which we will use for storing incoming messages. We will also create another machine where we will set up Prometheus for monitoring the application.
The virtual machines will be configured using Ansible, so before running Terraform, please ensure you have one of the latest versions of Ansible installed. Also, install the necessary roles from Ansible Galaxy:
vozerov@mba:~/events/terraform (master) $ cd ../ansible/
vozerov@mba:~/events/ansible (master) $ ansible-galaxy install -r requirements.yml
- cloudalchemy-prometheus (master) is already installed, skipping.
- cloudalchemy-grafana (master) is already installed, skipping.
- sansible.kafka (master) is already installed, skipping.
- sansible.zookeeper (master) is already installed, skipping.
- geerlingguy.docker (master) is already installed, skipping.
vozerov@mba:~/events/ansible (master) $Inside the ansible folder, there is an example configuration file .ansible.cfg that I use. It might be useful.
Before creating the virtual machines, ensure that your ssh-agent is running and your ssh key is added, otherwise Terraform won't be able to connect to the created machines. I, of course, ran into a bug on OS X: . To avoid running into such an issue, before starting Terraform, add a small variable to your env:
vozerov@mba:~/events/terraform (master) $ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESIn the folder with Terraform, we create the necessary resources:
vozerov@mba:~\/events\/terraform (master) $ terraform apply -target yandex_compute_instance.build -target yandex_compute_instance.monitoring -target yandex_compute_instance.kafka
yandex_vpc_network.internal: Refreshing state... [id=enp2g2rhile7gbqlbrkr]
data.yandex_compute_image.ubuntu_image: Refreshing state...
yandex_vpc_subnet.internal-a: Refreshing state... [id=e9b1dad6mgoj2v4funog]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
... skipped ...
Plan: 3 to add, 0 to change, 0 to destroy.
... skipped ...If everything went well (as it should), we will have three virtual machines:
- build — the machine for testing and building the application. Docker was automatically installed by ansible.
- monitoring — the machine for monitoring — it has prometheus & grafana installed. The login/password is default: admin/admin
- kafka — a small machine with kafka installed, accessible on port 9092.
Let's make sure they are all in place:
vozerov@mba:~\/events (master) $ yc compute instance list
+----------------------+------------+---------------+---------+---------------+-------------+
| ID | NAME | ZONE ID | STATUS | EXTERNAL IP | INTERNAL IP |
+----------------------+------------+---------------+---------+---------------+-------------+
| fhm081u8bkbqf1pa5kgj | monitoring | ru-central1-a | RUNNING | 84.201.159.71 | 172.16.1.35 |
| fhmf37k03oobgu9jmd7p | kafka | ru-central1-a | RUNNING | 84.201.173.41 | 172.16.1.31 |
| fhmt9pl1i8sf7ga6flgp | build | ru-central1-a | RUNNING | 84.201.132.3 | 172.16.1.26 |
+----------------------+------------+---------------+---------+---------------+-------------+ The resources are in place, and from here we can extract their IP addresses. From now on, I will use the IP addresses for SSH connections and application testing. If you have an account on Cloudflare connected to Terraform, feel free to use the newly created DNS names.
By the way, when creating a virtual machine, an internal IP and an internal DNS name are provided, so we can refer to the servers within the network by names:
ubuntu@build:~$ ping kafka.ru-central1.internal
PING kafka.ru-central1.internal (172.16.1.31) 56(84) bytes of data.
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=1 ttl=63 time=1.23 ms
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=2 ttl=63 time=0.625 ms
^C
--- kafka.ru-central1.internal ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.625/0.931/1.238/0.308 msThis will be useful for specifying the endpoint for the application with Kafka.
Building the application
Great, we have the servers and the application — now all that's left is to build and publish it. We will use the regular docker build for compiling, and for the image repository, we will use Yandex's container registry service. But let's take it step by step.
Copy the application to the build machine, SSH into it, and build the image:
vozerov@mba:~\/events\/terraform (master) $ cd ..
vozerov@mba:~\/events (master) $ rsync -av app\/ ubuntu@84.201.132.3:app\/\n\n... skipped ...\n\nsent 3849 bytes received 70 bytes 7838.00 bytes\/sec
total size is 3644 speedup is 0.93\n\nvozerov@mba:~\/events (master) $ ssh 84.201.132.3 -l ubuntu
ubuntu@build:~$ cd app
ubuntu@build:~\/app$ sudo docker build -t app .
Sending build context to Docker daemon 6.144kB
Step 1\/9 : FROM golang:latest AS build
... skipped ...\n\nSuccessfully built 9760afd8ef65
Successfully tagged app:latestHalf the work is done — now we can check the functionality of our application by running it and pointing it at kafka:
ubuntu@build:~\/app$ sudo docker run --name app -d -p 8080:8080 app \/app\/app -kafka=kafka.ru-central1.internal:9092\n\nFrom the local machine, we can send a test event and see the response:\n\nvozerov@mba:~\/events (master) $ curl -D - -s -X POST -d '{"key1":"data1"}' http:\/\/84.201.132.3:8080\/post
HTTP\/1.1 200 OK\nContent-Type: application\/json\nDate: Mon, 13 Apr 2020 13:53:54 GMT\nContent-Length: 41\n\n{"status":"ok","partition":0,"Offset":0}\nvozerov@mba:~\/events (master) $The application responded with a success message and provided the id of the partition and offset where the message was recorded. The next step is to create a registry in Yandex.Cloud and upload our image there (how to do this in three lines is described in the registry.tf file). Let's create the storage:
vozerov@mba:~\/events\/terraform (master) $ terraform apply -target yandex_container_registry.events\n\n... skipped ...\n\nPlan: 1 to add, 0 to change, 0 to destroy.\n\n... skipped ...\n\nApply complete! Resources: 1 added, 0 changed, 0 destroyed.There are several methods for authentication in the container registry — using an oauth token, an iam token, or a service account key. More details about these methods can be found in the documentation. We will use the service account key, so let's create the account:
vozerov@mba:~\/events\/terraform (master) $ terraform apply -target yandex_iam_service_account.docker -target yandex_resourcemanager_folder_iam_binding.puller -target yandex_resourcemanager_folder_iam_binding.pusher\n\n... skipped ...\n\nApply complete! Resources: 3 added, 0 changed, 0 destroyed.Now we just need to create a key for it:
vozerov@mba:~\/events\/terraform (master) $ yc iam key create --service-account-name docker -o key.json
id: ajej8a06kdfbehbrh91p
service_account_id: ajep6d38k895srp9osij
created_at: "2020-04-13T14:00:30Z"
key_algorithm: RSA_2048We obtain information about the id of our storage, transfer the key, and authenticate:
vozerov@mba:~\/events\/terraform (master) $ scp key.json ubuntu@84.201.132.3:\nkey.json 100% 2392 215.1KB\/s 00:00\n\nvozerov@mba:~\/events\/terraform (master) $ ssh 84.201.132.3 -l ubuntu\n\nubuntu@build:~$ cat key.json | sudo docker login --username json_key --password-stdin cr.yandex\nWARNING! Your password will be stored unencrypted in \/home\/ubuntu\/.docker\/config.json.\nConfigure a credential helper to remove this warning. See\nhttps:\/\/docs.docker.com\/engine\/reference\/commandline\/login\/#credentials-store\n\nLogin Succeeded\nubunt@build:~$To upload the image to the registry, we will need the ID of the container registry, which we will get from the yc utility:
vozerov@mba:~ $ yc container registry get events
id: crpdgj6c9umdhgaqjfmm
folder_id:
name: events
status: ACTIVE
created_at: "2020-04-13T13:56:41.914Z"After that, we tag our image with a new name and upload it:
ubuntu@build:~$ sudo docker tag app cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
ubuntu@build:~$ sudo docker push cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
The push refers to repository [cr.yandex/crpdgj6c9umdhgaqjfmm/events]
8c286e154c6e: Pushed
477c318b05cb: Pushed
beee9f30bc1f: Pushed
v1: digest: sha256:1dd5aaa9dbdde2f60d833be0bed1c352724be3ea3158bcac3cdee41d47c5e380 size: 946We can confirm that the image has been successfully uploaded:
vozerov@mba:~/events/terraform (master) $ yc container repository list
+----------------------+-----------------------------+
| ID | NAME |
+----------------------+-----------------------------+
| crpe8mqtrgmuq07accvn | crpdgj6c9umdhgaqjfmm/events |
+----------------------+-----------------------------+By the way, if you install the yc utility on a Linux machine, you can use the command
yc container registry configure-dockerto configure Docker.
Conclusion
We have done a lot of hard work and as a result:
- We have designed the architecture of our future service.
- We have written an application in Go that implements our business logic.
- We built it and released it into a private container registry.
In the next part, we'll move on to the exciting part — we will release our application into production and finally put it under load. Stay tuned!
This material is available in the video recording of the open workshop REBRAIN & Yandex.Cloud: Handling 10,000 requests per second in Yandex Cloud —
If you're interested in attending such events online and asking questions in real-time, join us at.
We would like to extend a special thanks to Yandex.Cloud for making this event possible. Here’s the link to them —
If you need to migrate to the cloud or have questions about your infrastructure, .
P.S. We have 2 free audits per month; your project might be one of them.
Source: habr.com
