At the end of last year, another live broadcast of the Russian PostgreSQL community took place , where its co-founder Nikolai Samokhvalov spoke with the technical director of "Flanta," Dmitry Stolyarov, about this DBMS in the context of Kubernetes.
We publish the transcript of the main part of this discussion, and on the full video recording is available:
Databases and Kubernetes
NS: We're not going to talk about VACUUM and CHECKPOINTS today. We want to discuss Kubernetes. I know you have many years of experience. I've watched your videos and even rewatched some parts... Let's get straight to the point: why use Postgres or MySQL in K8s?
DS: There is no definitive answer to this question, nor can there be. But generally, it’s about simplicity and convenience... potential ones. Everyone wants managed services, after all.
NS: Like , but on your own?
DS: Yes: like RDS, but anywhere.
NS: "Anywhere"—that’s a good point. In large companies, everything is located in different places. So then, if it’s a big company, why not use a ready-made solution? For example, Nutanix has its own developments, and other companies (like VMware…) offer the same "RDS, but on your own."
DS: But we’re talking about a specific implementation that will only work under certain conditions. When it comes to Kubernetes, there’s a vast variety of infrastructure (that can be in K8s). Essentially, it’s a standard for cloud API...
NS: And it's free!
DS: That’s not so important. Free services matter to only a small segment of the market. What’s more important is... You probably remember the presentation...»?
NS: Yes.
DS: I realized that it was received very ambiguously. Some people thought I was saying, "Guys, let’s move all databases to Kubernetes!" while others concluded that it was all just dreadful bicycles. But what I actually wanted to say was: "Look at what’s happening, what problems there are, and how they can be solved. Should you move databases to Kubernetes now? For production? Only if you enjoy... doing certain things. But for dev, I can say I recommend it. The dynamic creation/deletion of environments is crucial for dev."
NS: By 'dev,' do you mean all environments that are not prod? Staging, QA...
DS: When we talk about perf-stands, probably not, because their requirements are specific. If we talk about special cases where a very large database is needed on staging, then probably not either… If it's a static, long-lived environment, what's the benefit of having the database in K8s?
NS: None. But where do we see static environments? Static environments are already outdated as of tomorrow.
DS: Staging can be static. We have clients…
NS: Yes, I have them too. It's a big problem if you have a database of 10 TB, and staging is 200 GB…
DS: I have a really cool case! The staging environment has the production database, which is being modified. There's a button: “deploy to production.” These changes — deltas — are synchronized (seem to sync just via API) into production. This is a very exotic option.
NS: I've seen startups in the Valley that are still on RDS or even on Heroku — this is a story from 2-3 years ago — and they download a dump to their laptop. Because the database is only 80 GB so far, and there's space on the laptop. Later, they buy disks for each one to have 3 databases to conduct different developments. This happens too. I’ve also seen that they aren’t afraid to copy prod to staging — it really depends on the company. But I've seen both fear and that often there's not enough time or hands. But before we move on to this topic, I would like to hear about Kubernetes. Am I correct in understanding that no one has it in prod yet?
DS: We have small databases in prod. We're talking about sizes in tens of gigabytes and non-critical services for which it was too much hassle to make replicas (and there’s not much need either). And provided that there's decent storage under Kubernetes. This database was running on a virtual machine — conditionally in VMware, atop SAN. We placed it in and now we can move it from machine to machine.
NS: Databases of this size, up to 100 GB, on good disks and with a good network can be rolled out in just a few minutes, right? A speed of 1 GB per second is no longer exotic.
DS: Yes, for a linear operation that's not a problem.
NS: Okay, we should only be thinking about prod. And if we are considering Kubernetes for non-prod environments — how to do it? I see that Zalando , Crunchy , there are some other options. And there's — this is our good acquaintance Alvaro from Spain: they are essentially creating not just , but an entire distribution (), which decided to include not only Postgres itself but also a backup and the Envoy proxy…
DS: Envoy for what purpose? Load balancing specifically for Postgres traffic?
NS: Yes. So they see it this way: if you take a Linux distribution and the kernel, the standard PostgreSQL is the kernel, and they want to create a distribution that is cloud-friendly and can run on Kubernetes. They are integrating components (backups, etc.) and fine-tuning them so they work well.
DS: Very cool! Essentially, this is software to create your managed Postgres.
NS: Linux distributions have perpetual problems: how to create drivers that support all hardware. Their idea is that they will work in Kubernetes. I know that in the Zalando operator we recently saw a tie to AWS, and that's not very good. There shouldn’t be a tie to specific infrastructure—what’s the point then?
DS: I don’t know in what specific situation Zalando tied in, but in Kubernetes, the storage is currently structured so that you can’t take a disk backup in a generic way. Recently, in the latest version of the standard— —they made it possible to take snapshots, but where is it implemented? Honestly, it’s still so raw… We are trying CSI on top of AWS, GCE, Azure, vSphere, but as soon as you start using it, you see that it's not yet ready.
NS: That's why sometimes you have to tie it to infrastructure. I think this is still an early stage—growth problems. The question is: what would you advise newcomers who want to try PgSQL in K8s? Which operator, perhaps?
DS: The problem is that Postgres is 3% for us. We have a very large list of different software in Kubernetes; I won’t even list everything. For example, Elasticsearch. There are plenty of operators: some are actively developing, others are not. We have set requirements for what should be in an operator for us to take it seriously. An operator specifically for Kubernetes—not an 'operator to do something in the context of Amazon'… In fact, we use a single operator quite widely (= almost all clients)— (we will soon publish an article about it too).
NS: But there is none for MySQL either? I know that Percona… since they are now dealing with MySQL, MongoDB, and Postgres, they should come up with some universal solution: for all databases, for all cloud providers.
DS: We haven't gotten to the operators for MySQL. It's not our main focus right now. MySQL works fine in standalone mode. Why use an operator if you can just run the database… You can run a Docker container with PostgreSQL, or you can run it the straightforward way.
NS: That was a question too. Completely without an operator?
DS: Yes, 100% of the time we have PostgreSQL running without an operator. For now, that's how it is. We actively use an operator for Prometheus and Redis. We plan to find an operator for Elasticsearch—this is the most urgent requirement because we want to deploy it in Kubernetes in 100% of cases. Just as we want to consistently deploy MongoDB in Kubernetes as well. There are certain expectations here—there's a sense that something can be done in these cases. As for PostgreSQL, we haven't looked into it even. Of course, we know about various options, but in practice, we have it in standalone mode.
Database for testing in Kubernetes
NS: Let's move on to the topic of testing. How to deploy changes in the database from a DevOps perspective. There are microservices, multiple databases, and things are changing all the time. How to ensure proper CI/CD so that everything is orderly from the database perspective? What's your approach?
DS: There can't be just one answer. There are several parameters. The first is the size of the database we want to roll out. You mentioned that companies have different approaches when it comes to making a copy of the production database available on dev and stage.
NS: And under GDPR, I think they are becoming increasingly cautious… I can say that in Europe, fines have already started to be imposed.
DS: But often, software can be written to create a dump from production and obfuscate it. You end up with production data (snapshot, dump, binary copy…), but it's anonymized. Instead, there can be generation scripts: these can be fixtures or simply a script that generates a large database. The issue is: how much time does it take to create the base image? And how long does it take to deploy it in the required environment?
We've come up with a scheme: if the client has a fixture data set (a minimal version of the database), we use that by default. If it's about review environments, when we've created a branch, we spin up an instance of the application, and we roll out a small database there. But a good alternative also emerged. , when we take a dump from production once a day (at night) and build a Docker container with PostgreSQL and MySQL based on this data. If we need to deploy the database 50 times from this image, it can be done quite simply and quickly.
NS: By simple copying?
DS: The data is stored right in the Docker image. That is, we have a ready image, let’s say it’s 100 GB. Thanks to layers in Docker, we can quickly deploy this image as many times as needed. It's a straightforward method, but it works quite well.
NS: Next, when you test, it changes right inside Docker, right? Copy-on-write inside Docker — we discard and start again, everything's good. Great! And you're already using this extensively?
DS: For a while now.
NS: We're doing very similar things. Only we're not using Docker's copy-on-write, but something else.
DS: It's not generic. The Docker one works everywhere.
NS: In theory, yes. But we also have modules there; different modules can be made to work with different file systems. Here's the thing: we see all this from PostgreSQL's perspective. Now I've looked at it from Docker's side and saw that everything works on your end. But if the database is huge, say, 1 TB, then it all takes a long time: both the operations at night and pushing everything into Docker... Or is it all fine?
DS: What difference does it make? They are just blobs, simply bits and bytes.
NS: The difference is: do you do this through dump and restore?
DS: Not necessarily. There can be various methods for generating this image.
NS: For some clients, we have set it up so that instead of regularly generating a base image, we keep it constantly updated. Essentially, it acts as a replica, but it doesn't get data directly from the master, but through an archive. A binary archive, where WALs are applied each day, and backups are made... These WALs then make their way — with a slight delay (literally 1-2 seconds) — to the base image. From there, we clone by any means — currently by default we use ZFS.
DS: But with ZFS you're limited to one node.
NS: Yes. But ZFS has another magical : with it you can send a snapshot and even (I haven't tested this much, but…) you can send the delta between two PGDATA. In fact, we have another tool that we haven't really considered for such tasks. PostgreSQL has , functioning like a “smart” rsync, skipping much that doesn’t need to be reviewed because there hasn’t been any change. We can perform a quick synchronization between two servers and roll back just as easily.
So, we aim to create a tool from this more DBA-like perspective that allows the same thing you mentioned: we have one database, but we want to test something 50 times, almost simultaneously.
DS: 50 times means you need to order 50 Spot instances.
NS: No, we do it all on one machine.
DS: But how will you deploy 50 times if this single database is, say, terabytes? It likely needs, let's say, 256 GB of RAM?
NS: Yes, sometimes a lot of memory is needed — that’s normal. But here’s a real-life example. On the production machine, there are 96 cores and 600 GB. At the same time, 32 cores are used for the database (even 16 cores sometimes now) and about 100-120 GB of memory.
DS: And it fits 50 copies?
NS: Well, it’s a single copy; then it works copy-on-write (ZFS)… I’ll explain in more detail.
For instance, we have a 10 TB database. We created the disk for it, and ZFS further compressed its size by about 30-40%. Since we don’t perform load testing, the exact response time isn't crucial: it’s fine if it’s up to 2 times slower.
We allow developers, QA, DBAs, etc., to perform testing in 1-2 streams. For example, they can initiate some migration. It doesn’t require 10 cores right away — it needs 1 Postgres backend, 1 core. The migration will start — maybe, will also start, then the second core will be engaged. We have 16-32 cores allocated, so 10 people can work simultaneously without any issues.
Since physically PGDATA it’s the same, it turns out that we are actually fooling Postgres. The trick is: for example, 10 Postgres instances are started simultaneously. What’s the usual problem? They set , let’s say, at 25%. Thus, it’s 200 GB. You can’t start more than three like this because memory will run out.
But at some point, we realized this isn’t necessary: we set shared_buffers at 2 GB. PostgreSQL has an , and in reality, only it affects the . We set that to 0.5 TB. And it doesn’t even matter that they aren't actually there: it builds plans as if they are.
Accordingly, when we test any migration, we can gather all the plans — we will see how it will happen in production. The seconds will be different (slower), but the data we actually read and the plans themselves (such as which JOINs, etc.) will be exactly the same as in production. Additionally, you can run multiple such checks on one machine.
DS: Don't you think there are several problems here? The first is that this solution only works on PostgreSQL. This approach is very specific; it is not generic. The second is that Kubernetes (and everything that cloud technologies are now moving towards) assumes many nodes, and these nodes are ephemeral. In your case, it’s stateful, a persistent node. These things cause contradictions for me.
NS: First, I agree, this is purely a Postgres story. I think if we have any direct I/O and a buffer pool that uses almost all the memory, this approach won't work — the plans will be different. But for now, we're only dealing with Postgres and not thinking about others.
Regarding Kubernetes. You keep telling everyone that we have a persistent database. If an instance crashes, the main thing is to save the disk. So we also have the entire platform in Kubernetes, with the Postgres component separated (although it will be there at some point). So it's like this: the instance crashed, but we saved its PV and simply connected it to another (new) instance, as if nothing had happened.
DS: From my perspective, we create pods in Kubernetes. K8s is elastic: nodes are ordered as needed. The task is simply to create a pod and specify that it needs X resources, and then K8s will handle the rest. However, storage support in Kubernetes is still unstable: in , in (this release came out weeks ago) these features are still in beta.
In six months to a year, it will become more or less stable, or at least declared as such. Then, the ability to take snapshots and resize will fully resolve your task. Because you have a database. Yes, it may not be very fast, but speed depends on what’s 'under the hood', as some implementations can handle copying and copy-on-write at the disk subsystem level.
NS: Here, it also needs all the engines (Amazon, Google...) to start supporting this version — that also takes some time.
DS: For now, we are not using them. We are using our own.
Local development under Kubernetes.
NS: Have you ever encountered the desire to run all pods on a single machine and perform a small test? Just to quickly get a proof of concept and check if the application works in Kubernetes without allocating a bunch of machines for it. There’s Minikube, right?
DS: It seems to me that this case of deploying on a single node is solely about local development. Or some manifestations of such a pattern. There’s also , there’s , . We're moving towards using Kubernetes IN Docker. We’ve just started working with it for tests.
NS: I used to think that this was an attempt to wrap all pods in a single Docker image. But it turned out to be something entirely different. There are still separate containers, separate pods — just in Docker.
DS: Yes. And they’ve made a rather amusing simulation, but the point is... We have a deployment utility — . We want to create a mode in it — conditionally werf up: “Set up my local Kubernetes.” And then launch a conditional werf follow. Then the developer can edit in the IDE while a process is running in the system that sees changes, rebuilds images, and redeploys them in the local K8s. This is how we want to tackle the problem of local development.
Snapshots and cloning databases in the realm of K8s
NS: If we return to copy-on-write. I noticed that clouds also have snapshots. They work differently. For example, in GCP: you have a multi-terabyte instance on the East Coast of the USA. You periodically take snapshots. You spin up a copy of the disk from the snapshot on the West Coast — in just a few minutes it’s ready, and it works very quickly, you just need to fill the cache in memory. But these clones (snapshots) are to 'provision' a new volume. It's great when you need to create many instances.
However, for tests, it seems to me that the snapshots you mention in Docker or those I mention in ZFS, btrfs, and even LVM… — they allow you not to create real new data on a single machine. In the cloud, you'll be paying for them each time and waiting not seconds, but minutes (and in the case of , possibly even hours).
Instead, you can get this data in a second or two, run the test, and discard it. These snapshots solve different tasks. In the first case — to scale up and obtain new replicas, and in the second — for testing.
DS: I disagree. Properly implementing volume cloning is a cloud task. I haven't looked at their implementation, but I know how we do it on hardware. We have Ceph, where you can say to any physical volume () clone and get a second volume with the same characteristics in just tens of milliseconds, etc. One must understand that it uses an intricate copy-on-write mechanism. Why can't the cloud do the same? I'm sure they are trying to implement it one way or another.
NS: But they will still take seconds, tens of seconds, to spin up an instance, set up Docker, etc.
DS: Why do we need to spin up an entire instance? We have instances with 32 cores, 16 cores… and they can handle a certain number of workloads—say, four. When we request a fifth, an instance will start, and then it will be deleted.
NS: Yes, interestingly, in Kubernetes, there’s a different narrative. Our database is not in K8s, and it's a single instance. However, cloning a multi-terabyte database takes no more than two seconds.
DS: That's cool. But my initial point is that this is not a generic solution. Yes, it's great, but it only works for Postgres and only on one node.
NS: It's not just for Postgres: these plans, as I described, will only work that way for it. But if we don't worry about plans, and we just need all the data for functional testing, then it can work with any DBMS.
DS: Many years ago, we did something similar with LVM snapshots. That's classic. That approach was very actively used. The downside of stateful nodes — it's a hassle. Because you can't let them drop, you always have to keep track of them…
NS: Do you see any possibility of a hybrid here? Suppose a stateful pod is running for several users (many testers). We have one volume, but thanks to the file system, the clones are local. If a pod crashes, the disk remains — the pod will restart, read all the cloning information, bring everything back, and say: 'Here are your clones running on these ports, continue working with them.'
DS: Technically, this means that within Kubernetes, it's one pod, inside which we run multiple Postgres instances.
NS: Yes. It has a limit: for instance, no more than 10 people can work with it at once. If 20 are needed, we will spin up a second such pod. It’s completely feasible to clone it, resulting in a second full volume with the same 10 'thin' clones. Do you not see that possibility?
DS: Security questions need to be added here. This type of organization implies that this pod has high privileges (capabilities), as it can perform non-standard operations on the file system... But I repeat: I believe that in the medium term, Kubernetes will fix storage issues, and cloud services will resolve all matters related to volumes — everything will just work. There will be resizing, cloning... A volume exists — we say: 'Create a new one based on that,' and in a second and a half, we get what we need.
NS: I don't believe in a second and a half for many terabytes. With Ceph, you do it yourself, but you're talking about cloud services. Go to the cloud, create a clone of an EBS volume of many terabytes on EC2, and see what kind of performance you'll get. It won't take a few seconds. I'm very interested in when they'll reach that level. I understand what you're talking about, but I allow myself to disagree.
DS: Okay, but I said in the medium term, not short term. Within a few years.
About the PostgreSQL operator from Zalando
During the middle of this meeting, Alexey Klyukin, a former developer from Zalando, also joined in to tell the story of the PostgreSQL operator:
It's great that this topic is even being touched on: both Postgres and Kubernetes. When we started working on it at Zalando in 2017, it was a topic that everyone wanted to engage with, but no one was doing it. Everyone was already using Kubernetes, but when they asked how to deal with databases, even people like , who preached K8s, said something like this:
"Go for managed services and use them, don't run databases in Kubernetes. Otherwise, your K8s might decide to upgrade, take down all nodes, and your data will disappear far, far away."
We decided to create an operator that would, against this advice, run Postgres databases in Kubernetes. And we had a good foundation — . This is automatic failover for PostgreSQL done correctly, i.e., using etcd, consul, or ZooKeeper as a cluster information storage. A storage that will provide everyone who asks the same information about who the current leader is — despite our distributed nature — to avoid split brain. Additionally, we had for it.
In general, the need for auto failover arose for the company after migrating from an internal hardware data center to the cloud. The cloud was built on a proprietary PaaS (Platform-as-a-Service) solution. It is open source, but it required significant effort to set up. It was called .
Initially, there was no Kubernetes. To be precise, when the proprietary solution was deployed, K8s was already available, but it was so immature that it wasn't suitable for production. This was around 2015 or 2016. By 2017, Kubernetes had become sufficiently mature — the need for migration there arose.
We already had a Docker container. There was a PaaS that used Docker. Why not try K8s? Why not write our own operator? Murat Kabilov, who joined us from Avito, started this as a project on his own initiative — just to 'play around,' — and the project 'took off.'
But overall, I wanted to talk about AWS. Why there was historically code related to AWS…
When you launch anything in Kubernetes, you need to understand that K8s is a work in progress. It is constantly evolving, improving, and sometimes even breaking. You have to pay close attention to all the changes in Kubernetes, be prepared to dive into it if necessary and understand how it works in detail — possibly more than you'd like. This is the case with any platform where you run your databases...
So, when we created the operator, we had Postgres working with an external volume (in this case, EBS, since we were working on AWS). The database grew, and at some point, a resize became necessary: for example, the original EBS size was 100 TB, the database grew to that size, and now we want to make the EBS 200 TB. How? You could dump/restore to a new instance, but that's slow and involves downtime.
Therefore, we wanted such a resize that would increase the EBS partition and then tell the file system to utilize the new space. And we did this, but at that time, Kubernetes had no API for resize operations. Since we were working on AWS, we wrote code for its API.
There's nothing stopping you from doing the same for other platforms. The operator doesn't have a dependency that it can only run on AWS; it will work on other platforms as well. In general, this is an open source project: if someone wants to speed up the adoption of the new API — you're welcome to do so. There are , pull requests — the Zalando team tries to respond to them promptly and promote the operator. As far as I know, the project in Google Summer of Code and some other similar initiatives. Zalando is actively working on it.
P.S. Bonus!
If you are interested in PostgreSQL and Kubernetes, we also want to draw your attention to the fact that last week the next Postgres Tuesday took place, where Nikolai spoke with Alexander Kukushkin from Zalando. The video from it is available .
P.P.S.
Also read in our blog:
- «»;
- «»;
- «»;
- «».
Source: habr.com
