At the end of May we held an online meetup on the topic of . We talked about containers, Kubernetes, and orchestration in general, selection criteria for infrastructure, and much more. Participants shared case studies from their own practices.
Participants:
- Yevgeny Potapov, CEO of "ITSumma". More than half of his clients are either already transitioning or want to transition to Kubernetes.
- Dmitry Stolyarov, CTO of "Flant". He has over 10 years of experience working with container systems.
- Denis Remchukov (aka Eric Oldmann), COO of argotech.io, ex-RAO EES. He promised to share case studies from the “bloody” enterprise.
- Andrei Fedorovsky, CTO of "News360.com"After the company was acquired by another player, he is responsible for several ML and AI projects and for the infrastructure.
- Ivan Kruglov, systems engineer, ex–Booking.com.The very person who has done a lot with Kubernetes by hand.
Topics:
- Insights from participants on containers and orchestration (Docker, Kubernetes, etc.); what they have tried in practice or analyzed.
- Case Study: The company is developing an infrastructure development plan for years. How is the decision made to build (or transition the current) infrastructure on containers and Kubernetes or not?
- Challenges in the cloud-native world, what is lacking, let’s fantasize about what tomorrow will bring.
An interesting discussion ensued, and the opinions of the participants were so diverse and generated so many comments that we want to share them with you. There is , and below is a summary of the discussion.
Is Kubernetes already a standard or just excellent marketing?
“We came to it (Kubernetes — Ed.) when no one knew about it yet. We came to it even before it existed. We wanted it before that” — Dmitry Stolyarov
Photo from Reddit.com
5-10 years ago, there were a huge number of tools, and no single standard. Every six months, a new product would appear, or even more than one. First Vagrant, then Salt, Chef, Puppet,… “and every six months you’re restructuring your infrastructure. You have five admins who are constantly busy rewriting configurations” — recalls Andrei Fedorovsky. He believes that Docker and Kubernetes "crushed" the others. Docker became the standard over the past five years, Kubernetes over the last two years. And that's good for the industry..
Dmitry Stolyarov and his team love Kubernetes. They wanted such a tool before it even existed and came to it when no one else knew about it. Currently, for the sake of convenience, they do not take on a client if they understand that they will not implement Kubernetes for them. At the same time, according to Dmitry, the company has numerous gigantic success stories of transforming terrible legacy systems.
Kubernetes is not just container orchestration; it is a configuration management system with an advanced API, a networking component, L3 load balancing, and Ingress controllers that allows for relatively easy resource management, scaling, and abstraction from the lower layers of infrastructure.
Unfortunately, in our lives, everything comes at a cost. And this cost is significant, especially when discussing a transition to Kubernetes for a company with a developed infrastructure, according to Ivan Kruglov. He could easily work both in a company with traditional infrastructure and with Kubernetes. The main thing is to understand the company and market specifics. However, for Evgeny Potapov, who would generalize Kubernetes to any container orchestration tool, this question doesn’t arise.
Evgeny drew an analogy with the situation in the 1990s when object-oriented programming emerged as a way to program complex applications. At that time, debates continued, and new tools supporting OOP emerged. Then microservices surfaced as a means to move away from the monolithic concept. This, in turn, led to the advent of containers and their management tools. "I think that soon we will reach a time when the question of whether to write a small application in a microservices way won't even arise; it will be written as a microservice by default," he believes. Similarly, Docker and Kubernetes will become standard solutions without the need for choice over time.
The problem of databases is statelessness.
Photo by
Nowadays, there are many recipes for launching databases in Kubernetes. Even how to separate the part working with disk I/O from, hypothetically, the application part of the database. Is it possible that in the future databases will be modified enough to be packaged in a way where one part is orchestrated through Docker and Kubernetes, while the storage part is provided through separate software in another part of the infrastructure? Will databases change as a product?
This description seems similar to queue management, but the reliability and synchronization requirements in traditional databases are much higher, according to Andrey. The cache hit ratio in normal databases remains around 99%. If a worker fails, a new one starts, and the cache is 'warmed up' from scratch. As long as the cache is not warmed up, the worker operates slowly, meaning it cannot handle user load. Without user load, the cache does not warm up. This creates a vicious cycle.
Dmitry fundamentally disagrees, stating that quorums and sharding solve the problem. But Andrey insists that this solution does not fit everyone. In some situations, a quorum may work, but it adds additional load on the network. A NoSQL database is not suitable in all cases.
The participants of the meetup divided into two camps.
Denis and Andrey claim that everything that writes to disk — databases and more — is impossible to do in the current Kubernetes ecosystem. It is impossible to maintain the integrity and consistency of production data in Kubernetes. This is a fundamental feature. Solution: hybrid infrastructure.
Even modern cloud-native databases like MongoDB and Cassandra, or message queues like Kafka or RabbitMQ, require persistent data storage outside of Kubernetes.
Evgeny counters: "Databases in Kubernetes are a trauma tied to post-Soviet or enterprise issues related to the lack of Cloud Adoption in Russia." Small or medium companies in the West use Cloud. Using Amazon RDS is easier than dealing with Kubernetes on your own. In Russia, Kubernetes is used 'on-premise' and databases are moved into it while trying to eliminate the zoo of technologies.
Dmitry also disagreed with the assertion that no databases can be kept in Kubernetes: "Not all databases are the same. If you push a giant relational database into it — absolutely not. But if you push something small and cloud-native, which is mentally prepared for a semi-ephemeral existence, everything will be fine." Dmitry also mentioned that database management tools are not ready for either Docker or Kubernetes, which leads to significant challenges.
Ivan, for his part, believes that even if we abstract from the concepts of stateful and stateless, the ecosystem of enterprise solutions in Kubernetes is still not ready. It is difficult to meet the requirements of legislative and regulatory bodies with Kuber. For example, it's impossible to create an identity provision solution where strict guarantees of server identification are required, down to the hardware that is placed into the servers. This area is developing, but there is currently no solution.
The participants could not reach an agreement, so there will be no conclusions in this part. Let's instead provide a couple of practical examples.
Case 1: Cybersecurity of the 'mega regulator' with databases outside Kubernetes
In the case of an advanced cybersecurity system, the use of containers and orchestration allows for defense against attacks and intrusions. For example, in one mega regulator, Denis and his team implemented a combination of an orchestrator with a trained SIEM service that analyzes logs in real-time and identifies the attack, breach, or failure process. In the event of an attack, an attempt to upload something, or a ransomware virus intrusion, it uses the orchestrator to spin up containers with applications faster than they can become infected or faster than the attacker can strike.
Case 2: Partial migration of Booking.com databases to Kubernetes
At Booking.com, the main database is MySQL with asynchronous replication — there is a master and a whole hierarchy of slaves. By the time Ivan left the company, a project had been initiated to migrate the slaves that could be 'taken out' with a certain level of impact.
In addition to the main database, there is an installation of Cassandra with custom orchestration that was developed before Kubernetes became mainstream. There are no issues in that respect, but it has persistent storage on local SSDs. Remote storage, even within a single data center, is not used due to high latency issues.
The third class of databases is the search service of Booking.com, where each node of the service is a database. Attempts to migrate the search service to Kubernetes failed because each node has 60-80 GB of local storage, which is difficult to 'bring up' and 'warm up'.
As a result, the search engine was not migrated to Kubernetes, and Ivan does not think there will be new attempts anytime soon. The MySQL database was half-migrated: only the slaves that it was not risky to ‘take out’. Cassandra has 'settled in' excellently.
Choosing infrastructure as a task without a universal solution
Photo by
Let's assume we have a new company, or a company where part of the infrastructure is outdated. They are developing an infrastructure development plan for years. How is the decision made whether to build the infrastructure on containers and Kubernetes or not?
Companies that are battling for nanoseconds are excluded from the discussion. Healthy conservatism pays off in terms of reliability, but there are still companies that should consider new approaches.
Ivan: "I would definitely start a company in the cloud right now, simply because it's faster," though not necessarily cheaper. With the development of venture capitalism, startups don't have major issues with funding, and the main task is to capture the market.
Ivan believes that the maturity of the current infrastructure is a criterion for choice. If significant investments were made in the past and it works, there's no point in overhauling it. However, if the infrastructure is underdeveloped and there are issues with tools, security, and monitoring, it makes sense to look towards distributed infrastructure.
Taxes will need to be paid in any case, and Ivan would pay the one that would allow him to pay less in the future. "Because simply by riding on a train that is operated by others, I will get much further than if I took another train, where I would have to refuel myself." — says Ivan. When the company is new, and latency requirements are in the tens of milliseconds, Ivan would look towards 'operators' where traditional databases are currently being hosted. They set up a replication chain that switches automatically in case of failover, etc.
For a small company with a couple of servers in Kubernetes, it makes no sense, — asserts Andrey. But if it's planning to grow to hundreds of servers or more, then automation and resource management systems are needed. 90% of cases justify the costs, regardless of the load and resources. It makes sense for everyone, from startups to large companies with a million-strong audience, to gradually consider container orchestration products. "Yes, this is truly the future," — believes Andrey.
Denis outlined two main criteria — scalability and operational resilience. He will choose the tools that are best suited for the task. 'This could be a makeshift no-name solution running Nutanix Community Edition. It could also be a secondary tier in the form of an application on Kubernetes with a database on the backend that is replicated and has specified RTO and RPO parameters' (recovery time/point objectives — example).
Evgeny pointed out a potential problem with staffing. Currently, there aren't many top-notch specialists on the market who understand the 'guts' of systems. Indeed, if the chosen technology is outdated, it's difficult to hire anyone other than older, disillusioned people who are burnt out. However, other participants believe it's a matter of workforce preparation.
If posed with the choice of launching a small company on Public Cloud with databases in Amazon RDS or 'on-premise' with databases in Kubernetes, despite some drawbacks, the choice of the participants was Amazon RDS.
Since most of the meetup attendees are not from the 'brutal' enterprise sector, distributed solutions are something to strive for. Storage systems must be distributed, reliable, and create latency measured in milliseconds, at most in tens., - summarized Andrey.
Assessment of Kubernetes usage
Listener Anton Zhbankov posed a trap question to the Kubernetes proponents: how was the technical and economic justification chosen and conducted? Why Kubernetes, and not virtual machines, for example?
Photo by
Dmitry and Ivan answered him. In both cases, through trial and error, a sequence of decisions was made, resulting in both participants arriving at Kubernetes. Now, businesses are starting to independently develop software that makes sense to migrate to Kubernetes. This is not about classical third-party systems like 1C. Kubernetes is helpful when developers need to quickly make releases during uninterrupted Continuous Improvement.
Andrey's team tried to create a scalable cluster based on virtual machines. The nodes fell like dominoes, sometimes leading to cluster failures. 'Theoretically, it could be completed and maintained manually, but it's tedious. If there's a solution on the market that allows you to work out of the box, we gladly turn to it. And as a result, we transitioned.' - Andrey shares.
Standards for such analysis and calculations exist, but no one can say how accurate they are on real hardware in operation. It is also important to understand each tool and ecosystem for calculations, but that is impossible.
What awaits us
Photo by
As technologies evolve, more fragmented pieces emerge, and then a phase transition occurs, leading to a vendor that has killed enough "bubbles" to unify everything in a single tool.
Don’t you think a moment will come when a tool emerges that becomes as essential as Ubuntu has for the Linux world? Perhaps a unified containerization and orchestration tool will also include Kubernetes. It will become easier to build on-premise clouds.
Ivan responded: "Google is currently building Anthos — it is their bundled offering that deploys the cloud and includes Kuber, Service Mesh, monitoring — all the connectivity needed for microservices in an on-premise setting. We are almost in the future."
Denis also mentioned Nutanix and VMware with their vRealize Suite product, which can handle similar tasks without containerization.
Dmitry shared the opinion that reducing "pain" and lowering taxes are two areas where we should expect improvements.
In conclusion of the discussion, let’s highlight the following problems of modern infrastructure.
- Three participants immediately identified an issue with stateful applications.
- Various security support issues, including the likelihood that several versions of Python, application servers, and components will end up in Docker.
Overconsumption, which is better addressed in a separate meetup.
The training issue, as orchestration represents a complex ecosystem.
A common problem in the industry is using tools for improper purposes.The rest of the conclusions are for you to make. For now, there remains a feeling that the combination of Docker and Kubernetes is not easily becoming the "central" part of the system. For example, operating systems are installed on hardware first, which cannot be said for containers and orchestration. Perhaps in the future, operating systems and containers will merge with cloud management software.
Photo byTaking this opportunity, I want to say hello to my mom and remind her that we have a Facebook group. , channel , featuring interesting posts from various tech blogs. And my channel , where I discuss development management in product companies.
Source: habr.com
