Over the years, Pinterest has had 300 million users create more than 200 billion pins on over 4 billion boards. To serve this army of users and vast content base, the portal has developed thousands of services, ranging from microservices that a few CPUs can handle to massive monoliths running on an entire fleet of virtual machines. And then came the moment when the company turned its gaze to k8s. What attracted Pinterest to Kubernetes? You will find out in our translation of the latest article from .
So, hundreds of millions of users and hundreds of billions of pins. To cater to this army of users and extensive content base, we developed thousands of services, ranging from microservices manageable by a few CPUs to gigantic monoliths running on an entire fleet of virtual machines. Additionally, we have various frameworks that may also require CPU resources, memory, or access to input/output operations.
In supporting this zoo of tools, the development team faces several challenges:
- Engineers lack a unified way to launch their working environments. Stateless services, stateful services, and projects in active development are based on entirely different technology stacks. This has led to the creation of a whole training course for engineers and complicates the work of our infrastructure team significantly.
- Developers with their own fleets of virtual machines place a huge burden on internal administrators. As a result, even simple operations such as OS or AMI updates take weeks and months. This contributes to increased load in what would otherwise be completely mundane situations.
- Challenges in creating global infrastructure management tools on top of existing solutions. The situation is further complicated by the difficulty of identifying the owners of virtual machines. So, we don’t know if it’s safe to reclaim these resources to work on other areas of our infrastructure.
Container orchestration systems are a way to unify workload management. They pave the way for increased development speed and simplify infrastructure management, as all resources involved in the project are managed by a single centralized system.
Figure 1: Infrastructure Priorities (reliability, developer performance, and efficiency).
The Cloud Management Platform team at Pinterest got acquainted with K8s in 2017. By the first half of 2017, we had documented most of our production capabilities, including our APIs and all our web servers. After that, we conducted a careful evaluation of various container orchestration solutions, cluster building, and operation systems. By the end of 2017, we decided to adopt Kubernetes. It was flexible enough and widely supported in the developer community.
By now, we have created our own primary cluster bootstrapping tools based on Kops and migrated existing infrastructure components — such as networking, security, metrics, logging, identity management, and traffic — to Kubernetes. We also implemented a workload modeling system for our resource, the complexity of which is hidden from developers. Currently, we are focused on ensuring cluster stability, scaling it, and onboarding new clients.
Kubernetes: The Pinterest Journey
Getting started with Kubernetes at Pinterest as a platform that would be loved by our engineers brought many challenges.
As a large company, we invested significant resources into infrastructure tools. For example, the security tools that handle certificates and distribute keys, traffic control components, service discovery systems, visibility components, and logging and metrics transmission. All of this was assembled for a reason: we went through a normal path of trial and error, and therefore wanted to integrate all this into the new infrastructure on Kubernetes instead of reinventing the old wheel on a new platform. This approach generally simplified migration, as all application support already exists and doesn’t need to be built from scratch.
On the other hand, the forecasting models for loads within Kubernetes (such as deployments, jobs, and daemon sets) are insufficient for our project. These usability issues present significant barriers to transitioning to Kubernetes. For instance, we have heard service developers complain about the lack of or incorrect entry configurations. We have also encountered improper use of template engines, leading to the creation of hundreds of copies with identical specifications and tasks, resulting in nightmarish debugging problems.
It was also very challenging to maintain different versions within the same cluster. Imagine the complexity of client support when you have to operate across multiple versions of the same runtime environment, along with their respective problems, bugs, and updates.
Pinterest custom resources and controllers
To simplify the process of implementing Kubernetes for our engineers, as well as to streamline and accelerate our infrastructure, we have developed our own Custom Resource Definitions (CRDs).
CRDs provide the following functionalities:
- Combining various native Kubernetes resources to work as a single load. For example, the PinterestService resource includes a deployment, an ingress service, and a configuration map. This allows developers to not worry about DNS configuration.
- Implementing the necessary application support. The user only needs to focus on the container specification according to their business logic, while the CRD controller deploys all necessary init containers, environment variables, and pod specifications. This provides a fundamentally higher level of comfort for developers.
- CRD controllers also manage the lifecycle of their own resources and enhance debugging availability. This includes reconciling desired and actual specifications, updating CRD status, and maintaining event logs, among other tasks. Without CRDs, developers would have to manage a multitude of resources, increasing the likelihood of errors.
Here is an example of a PinterestService and an internal resource managed by our controller:
As mentioned above, to support the user container, we need to integrate it with the initialization container and several add-ons to ensure security, visibility, and interaction with network traffic. Additionally, we have created configuration map templates and implemented support for PVC templates for batch jobs, as well as tracking multiple environment variables to monitor identification, resource consumption, and garbage collection.
It's hard to imagine developers wanting to write these configuration files manually without CRD support, let alone maintain and debug configurations afterward.
Application Deployment Workflow
The diagram above illustrates how to deploy a custom Pinterest resource in a Kubernetes cluster:
- Developers interact with our Kubernetes cluster through the CLI and user interface.
- CLI/UI tools retrieve YAML configuration files for the workflow and other build properties (the same version ID) from Artifactory, and then send them to the Job Submission Service. This step ensures that only working versions are deployed in the cluster.
- JSS acts as a gateway for various platforms, including Kubernetes. This is where user authentication, quota issuance, and partial configuration validation of our CRD takes place.
- After validating the CRD on the JSS side, the information is sent to the k8s platform API.
- Our CRD controller monitors events on all custom resources. It transforms CRs into native k8s resources, adds necessary modules, sets appropriate environment variables, and performs other housekeeping tasks, thereby ensuring sufficient infrastructure support for containerized user applications.
- The CRD controller then passes the received data to the Kubernetes API for processing by the scheduler and to be put into operation.
Note: This pre-release deployment workflow was created for the early users of the new k8s platform. We are currently in the process of refining this process so that it fully integrates with our new CI/CD system. This means we cannot share everything related to Kubernetes yet. We look forward to sharing our experiences and discussing our team's progress in this area in our next blog post, “Building a CI/CD platform for Pinterest.”
Types of Special Resources
Based on Pinterest's specific needs, we have developed the following CRDs that fit various workflows:
- PinterestService is a long-established stateless service. Many of our core systems are built on a set of such services.
- PinterestJobSet models end-to-end batch jobs. At Pinterest, there is a common scenario where multiple jobs run the same containers in parallel, independent of other similar processes.
- PinterestCronJob is widely used in conjunction with small periodic loads. It is a wrapper around native cron functionality with support mechanisms in Pinterest responsible for security, traffic, logging, and metrics.
- PinterestDaemon includes infrastructure daemons. This family continues to grow as we add more support for our clusters.
- PinterestTrainingJob extends to TensorFlow and PyTorch processes, providing the same level of support during execution as all other CRDs. Since TensorFlow and other machine learning systems are heavily utilized at Pinterest, it made sense to build a separate CRD around them.
We are also working on PinterestStatefulSet, which will soon be adapted for data storage and other stateful systems.
Runtime Environment Support
When an application module starts in Kubernetes, it automatically receives a certificate for its own identification. This certificate is used to access the secret storage or to communicate with other services via mTLS. Meanwhile, the container initialization configurator and Daemon will load all necessary dependencies before launching the container application. Once everything is ready, the traffic sidecar and Daemon will register the module's IP address in our Zookeeper so that clients can discover it. All of this will work since the network module was configured even before the application started.
The examples above illustrate typical support for workloads during runtime. Other types of workloads may require slightly different support, but all are represented as pod-level sidecars, node-level, or Daemon-level virtual machines. We ensure that everything is deployed within the management infrastructure and coordinated between applications, significantly reducing the burden of technical work and customer support in the process.
Testing and QA
We have built an end-to-end testing pipeline on top of the existing Kubernetes testing infrastructure. These tests are applied across all our clusters. Our pipeline has undergone numerous revisions before becoming part of the product cluster.
In addition to testing systems, we have monitoring and alerting systems that constantly oversee the status of system components, resource consumption, and other important metrics, notifying us only when human intervention is needed.
Alternatives
We considered some alternatives for custom resources, such as mutation admission controllers and templating systems. However, all of them come with significant operational challenges, so we chose the CRD path.
The mutation admission controller was used to introduce sidecars, environment variables, and other support during execution. However, it faced various issues, such as resource binding and lifecycle management, which do not occur with CRDs.
Note: Template systems, like Helm charts, are also widely used for launching applications with similar configurations. However, our working applications are too diverse to manage them with templates. Additionally, too many errors will occur during continuous deployment when using templates.
Upcoming Work
Currently, we are dealing with a mixed workload across all our clusters. To support such varied processes in different types and sizes, we are focusing on the following areas:
- The cluster set distributes large applications across different clusters to ensure scalability and stability.
- Ensuring the stability, scalability, and visibility of the cluster to establish a connection between the application and its SLA.
- Managing resources and quotas so that applications do not conflict with each other, while the scale of the cluster is controlled from our side.
- A new CI/CD platform to support and deploy applications in Kubernetes.
Source: habr.com
