{"id":55467,"date":"2020-01-21T00:00:00","date_gmt":"2020-01-20T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/postgres-vtornik-5-postgresql-i-kubernetes-ci-cd-avtomatizatsiya-testirovaniya"},"modified":"2020-02-18T14:03:35","modified_gmt":"2020-02-18T11:03:35","slug":"postgres-vtornik-5-postgresql-i-kubernetes-ci-cd-avtomatizatsiya-testirovaniya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/postgres-vtornik-5-postgresql-i-kubernetes-ci-cd-avtomatizatsiya-testirovaniya","title":{"rendered":"Postgres Tuesday #5: \"PostgreSQL and Kubernetes. CI\/CD. Test Automation\"","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Postgres
\n
\nAt the end of last year, another live broadcast of the Russian PostgreSQL community took place #RuPostgres<\/a><\/noindex>, where its co-founder Nikolai Samokhvalov spoke with the technical director of \"Flanta,\" Dmitry Stolyarov, about this DBMS in the context of Kubernetes.<\/p>\n

We publish the transcript of the main part of this discussion, and on the community's YouTube channel<\/a><\/noindex> the full video recording is available:<\/a><\/noindex><\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center><\/p>\n

Databases and Kubernetes<\/h2>\n

\nNS<\/b>: We won't discuss VACUUM and CHECKPOINTs today. We want to talk about Kubernetes. I know you have many years of experience. I\u2019ve watched your videos and even rewatched some clips... Let\u2019s get straight to the point: why use Postgres or MySQL in K8s at all?<\/i><\/p>\n

DS<\/b>: There is no definitive answer to this question, nor can there be. But generally, it\u2019s about simplicity and convenience... potential ones. Everyone wants managed services, after all.<\/p>\n

NS<\/b>: Like RDS<\/a><\/noindex>, but on your own?<\/i><\/p>\n

DS<\/b>: Yes: like RDS, but anywhere.<\/p>\n

NS<\/b>: \"Anywhere\"\u2014that\u2019s a good point. In large companies, everything is located in different places. So then, if it\u2019s a big company, why not use a ready-made solution? For example, Nutanix has its own developments, and other companies (like VMware\u2026) offer the same \"RDS, but on your own.\"<\/i><\/p>\n

DS<\/b>: But we\u2019re talking about a specific implementation that will only work under certain conditions. When it comes to Kubernetes, there\u2019s a vast variety of infrastructure (that can be in K8s). Essentially, it\u2019s a standard for cloud API...<\/p>\n

NS<\/b>: And it's free!<\/i><\/p>\n

DS<\/b>: That\u2019s not so important. Free services matter to only a small segment of the market. What\u2019s more important is... You probably remember the presentation...Databases and Kubernetes<\/a><\/noindex>\u00bb?<\/p>\n

NS<\/b>: Yes.<\/i><\/p>\n

DS<\/b>: I realized that people received it very ambiguously. Some thought I was saying: \u201cGuys, let's move all databases to Kubernetes!\u201d, while others decided it was just nonsense. What I actually wanted to convey was something entirely different: \u201cLook at what's happening, what issues exist, and how we can solve them. Is it a good idea to run databases in Kubernetes? In production? Only if you enjoy... dealing with certain things. But for development \u2014 I can say I recommend it. For dev, the ability to dynamically create\/delete environments is crucial.\u201d<\/p>\n

NS: By dev, do you mean all environments that are not prod? Staging, QA\u2026<\/i><\/p>\n

DS<\/b>: When we talk about perf-stands, probably not, because their requirements are specific. If we talk about special cases where a very large database is needed on staging, then probably not either\u2026 If it's a static, long-lived environment, what's the benefit of having the database in K8s?<\/p>\n

NS<\/b>: None. But where do we see static environments? Static environments are already outdated as of tomorrow.<\/i><\/p>\n

DS<\/b>: Staging can be static. We have clients\u2026<\/p>\n

NS<\/b>: Yes, I have them too. It's a big problem if you have a database of 10 TB, and staging is 200 GB\u2026<\/i><\/p>\n

DS<\/b>: I have a really cool case! The staging area has a production database, which is modified, and there\u2019s a button: \u201cpromote to production\u201d. Those changes \u2014 the deltas \u2014 are pushed (it seems they sync simply via an API) into production. This is a very exotic option.<\/p>\n

NS<\/b>: I\u2019ve seen startups in the Valley still using RDS or even Heroku \u2014 these are stories from 2-3 years ago \u2014 and they download dumps to their laptops. Because the database is only 80 GB, and there\u2019s space on the laptop. Then they buy disks for everyone to hold 3 databases for different developments. It happens that way too. I've also seen that they are not afraid to copy prod to staging \u2014 it really depends on the company. But I\u2019ve seen some who are very afraid, and often there\u2019s not enough time or hands. But before we move on to that topic, I\u2019d like to hear about Kubernetes. Am I correct in my understanding that no one uses it in prod right now?<\/i><\/p>\n

DS<\/b>: We have small databases in prod. We\u2019re talking about volumes in the tens of gigabytes and non-critical services, for which it was too much trouble to create replicas (and there was no need either). And provided that Kubernetes has decent storage. This database worked on a virtual machine \u2014 so, in VMware, on top of storage. We placed it in PV<\/a><\/noindex> and now we can move it from machine to machine.<\/p>\n

NS<\/b>: Databases of this size, up to 100 GB, on good disks and with a good network can be rolled out in just a few minutes, right? A speed of 1 GB per second is no longer exotic.<\/i><\/p>\n

DS<\/b>: Yes, for a linear operation that's not a problem.<\/p>\n

NS<\/b>: Okay, we should only be thinking about prod. And if we are considering Kubernetes for non-prod environments \u2014 how to do it? I see that Zalando is making an operator<\/a><\/noindex>, Crunchy is developing<\/a><\/noindex>, there are some other options. And there's OnGres<\/a><\/noindex> \u2014 this is our good acquaintance Alvaro from Spain: they are essentially creating not just an operator<\/a><\/noindex>, but an entire distribution (StackGres<\/a><\/noindex>), where besides Postgres itself, we also decided to include backups, an Envoy proxy...<\/i><\/p>\n

DS<\/b>: Envoy for what purpose? Load balancing specifically for Postgres traffic?<\/p>\n

NS<\/b>: Yes. So they see it this way: if you take a Linux distribution and the kernel, the standard PostgreSQL is the kernel, and they want to create a distribution that is cloud-friendly and can run on Kubernetes. They are integrating components (backups, etc.) and fine-tuning them so they work well.<\/i><\/p>\n

DS<\/b>: Very cool! Essentially, this is software to create your managed Postgres.<\/p>\n

NS<\/b>: Linux distributions have perpetual problems: how to create drivers that support all hardware. Their idea is that they will work in Kubernetes. I know that in the Zalando operator we recently saw a tie to AWS, and that's not very good. There shouldn\u2019t be a tie to specific infrastructure\u2014what\u2019s the point then?<\/i><\/p>\n

DS<\/b>: I don\u2019t know in what specific situation Zalando tied in, but in Kubernetes, the storage is currently structured so that you can\u2019t take a disk backup in a generic way. Recently, in the latest version of the standard\u2014 CSI specification<\/a><\/noindex> \u2014they made it possible to take snapshots, but where is it implemented? Honestly, it\u2019s still so raw\u2026 We are trying CSI on top of AWS, GCE, Azure, vSphere, but as soon as you start using it, you see that it's not yet ready.<\/p>\n

NS<\/b>: That's why sometimes you have to tie it to infrastructure. I think this is still an early stage\u2014growth problems. The question is: what would you advise newcomers who want to try PgSQL in K8s? Which operator, perhaps?<\/i><\/p>\n

DS<\/b>: The problem is that Postgres accounts for only 3% for us. We have a very large list of various software in Kubernetes, and I won't even list everything. For example, Elasticsearch. There are many operators\u2014some are actively developed, and others are not. We have made our own requirements for what an operator must have to be taken seriously. The operator specifically for Kubernetes\u2014not for 'an operator to do something in Amazon's environment'... In fact, we use a single operator quite extensively (nearly all our clients)\u2014 for Redis<\/a><\/noindex> (we will soon publish an article about it too)<\/i>.<\/p>\n

NS<\/b>: But there is none for MySQL either? I know that Percona\u2026 since they are now dealing with MySQL, MongoDB, and Postgres, they should come up with some universal solution: for all databases, for all cloud providers.<\/i><\/p>\n

DS<\/b>: We haven't gotten to the operators for MySQL. It's not our main focus right now. MySQL works fine in standalone mode. Why use an operator if you can just run the database\u2026 You can run a Docker container with PostgreSQL, or you can run it the straightforward way.<\/p>\n

NS<\/b>: That was a question too. Completely without an operator?<\/i><\/p>\n

DS<\/b>: Yes, 100% of the time we have PostgreSQL running without an operator. For now, that's how it is. We actively use an operator for Prometheus and Redis. We plan to find an operator for Elasticsearch\u2014this is the most urgent requirement because we want to deploy it in Kubernetes in 100% of cases. Just as we want to consistently deploy MongoDB in Kubernetes as well. There are certain expectations here\u2014there's a sense that something can be done in these cases. As for PostgreSQL, we haven't looked into it even. Of course, we know about various options, but in practice, we have it in standalone mode.<\/p>\n

Database for testing in Kubernetes<\/h2>\n

\nNS<\/b>: Let's move on to the topic of testing. How to deploy changes in the database from a DevOps perspective. There are microservices, multiple databases, and things are changing all the time. How to ensure proper CI\/CD so that everything is orderly from the database perspective? What's your approach?<\/i><\/p>\n

DS<\/b>: There can't be just one answer. There are several parameters. The first is the size of the database we want to roll out. You mentioned that companies have different approaches when it comes to making a copy of the production database available on dev and stage.<\/p>\n

NS<\/b>: And under GDPR, I think they are becoming increasingly cautious\u2026 I can say that in Europe, fines have already started to be imposed.<\/i><\/p>\n

DS<\/b>: But often you can write software that takes a dump from production and obfuscates it. You get production data (a snapshot, a dump, a binary copy\u2026), but they are anonymized. Instead, there can be generation scripts: these can be fixtures or just a script that generates a large database. The problem is, how long does it take to create a basic image? And how long does it take to deploy it in the required environment?<\/p>\n

We've come up with a scheme: if the client has a fixture data set (a minimal version of the database), we use that by default. If it's about review environments, when we've created a branch, we spin up an instance of the application, and we roll out a small database there. But a good alternative also emerged. variant<\/a><\/noindex>, when we take a dump from production once a day (at night) and build a Docker container with PostgreSQL and MySQL based on it, with those loaded data. If we need to deploy the database from this image 50 times, it can be done quite simply and quickly.<\/p>\n

NS<\/b>: By simple copying?<\/i><\/p>\n

DS<\/b>: The data is stored right in the Docker image. That is, we have a ready image, let\u2019s say it\u2019s 100 GB. Thanks to layers in Docker, we can quickly deploy this image as many times as needed. It's a straightforward method, but it works quite well.<\/p>\n

NS<\/b>: Then, when you test, it changes right inside Docker, right? Copy-on-write inside Docker\u2014we throw it out and start over, everything\u2019s good. Great! And are you already using this extensively?<\/i><\/p>\n

DS<\/b>: For a while now.<\/p>\n

NS<\/b>: We are doing very similar things. However, we don't use Docker's copy-on-write; we use something else.<\/i><\/p>\n

DS<\/b>: It is not generic. But Docker's works everywhere.<\/p>\n

NS<\/b>: Ideally, yes. But we also have modules; different modules can be made to work with different file systems. Here\u2019s the thing: we view everything differently from Postgres's perspective. Now I've looked at it from Docker's side and see that everything works for you. But if the database is huge, for example, 1 TB, then everything takes much longer: both operations at night and putting everything into Docker... And if you have to put 5 TB into Docker... Or is everything fine?<\/i><\/p>\n

DS<\/b>: What difference does it make? They are just blobs, simply bits and bytes.<\/p>\n

NS<\/b>: The difference is: do you do this through dump and restore?<\/i><\/p>\n

DS<\/b>: Not necessarily. There can be various methods for generating this image.<\/p>\n

NS<\/b>: For some clients, we have arranged it so that instead of regular generation of the base image, we keep it constantly up to date. It essentially acts as a replica, but it obtains data not directly from the master, but through an archive. A binary archive where WALs are piled up every day, and backups are taken there as well\u2026 These WALs then travel \u2014 with a slight delay (literally 1-2 seconds) \u2014 to the base image. From this, we can clone in any way \u2014 by default, we use ZFS.<\/i><\/p>\n

DS<\/b>: But with ZFS you're limited to one node.<\/p>\n

NS<\/b>: Yes. But ZFS has another magical send<\/a><\/noindex>: with it you can send a snapshot and even (I haven't tested this much, but\u2026) you can send the delta between two PGDATA<\/code>. In fact, we have another tool that we haven't really considered for such tasks. PostgreSQL has pg_rewind<\/a><\/noindex>, functioning like a \u201csmart\u201d rsync, skipping much that doesn\u2019t need to be reviewed because there hasn\u2019t been any change. We can perform a quick synchronization between two servers and roll back just as easily.<\/i><\/p>\n

So, we try to create a tool from this more DBA side that allows us to do the same thing that you mentioned: we have one database, but we want to test something 50 times, almost simultaneously.<\/i><\/p>\n

DS<\/b>: 50 times means you need to order 50 Spot instances.<\/p>\n

NS<\/b>: No, we do it all on one machine.<\/i><\/p>\n

DS<\/b>: But how will you deploy 50 times if this single database is, say, terabytes? It likely needs, let's say, 256 GB of RAM?<\/p>\n

NS<\/b>: Yes, sometimes a lot of memory is needed \u2014 that\u2019s normal. But here\u2019s a real-life example. On the production machine, there are 96 cores and 600 GB. At the same time, 32 cores are used for the database (even 16 cores sometimes now) and about 100-120 GB of memory.<\/i><\/p>\n

DS<\/b>: And it fits 50 copies?<\/p>\n

NS<\/b>: So, there is only one copy, and then it works with copy-on-write (ZFS-based)\u2026 I'll explain in more detail.<\/i><\/p>\n

For instance, we have a 10 TB database. We created the disk for it, and ZFS further compressed its size by about 30-40%. Since we don\u2019t perform load testing, the exact response time isn't crucial: it\u2019s fine if it\u2019s up to 2 times slower.<\/i><\/p>\n

We provide the opportunity for programmers, QA, DBAs, etc., to perform testing in 1-2 streams. For example, they can start some migration. It doesn't require 10 cores immediately \u2014 it needs 1 backend of Postgres, 1 core. The migration will start \u2014 maybe, autovacuum<\/a><\/noindex> will also start, then the second core will be engaged. We have 16-32 cores allocated, so 10 people can work simultaneously without any issues.<\/i><\/p>\n

Since physically PGDATA<\/code> it turns out to be identical, so we are actually fooling Postgres. The trick is: for example, 10 Postgres instances are started simultaneously. What is usually the problem? They set up shared_buffers<\/a><\/noindex>, let\u2019s say, at 25%. Thus, it\u2019s 200 GB. You can\u2019t start more than three like this because memory will run out.<\/i><\/p>\n

But at some point, we realized this isn\u2019t necessary: we set shared_buffers at 2 GB. PostgreSQL has an effective_cache_size<\/a><\/noindex>, and in reality, only it affects the plans<\/a><\/noindex>. We set that to 0.5 TB. And it doesn\u2019t even matter that they aren't actually there: it builds plans as if they are.<\/i><\/p>\n

Accordingly, when we test some migration, we can gather all the plans \u2014 we will see how it will happen in production. The seconds there will be different (slower), but the data we actually read, and the plans themselves (what JOINs there are, etc.) turn out to be exactly the same as in production. And you can simultaneously run many such checks on one machine.<\/i><\/p>\n

DS<\/b>: Don't you think there are several problems here? The first is that this solution only works on PostgreSQL. This approach is very specific; it is not generic. The second is that Kubernetes (and everything that cloud technologies are now moving towards) assumes many nodes, and these nodes are ephemeral. In your case, it\u2019s stateful, a persistent node. These things cause contradictions for me.<\/p>\n

NS<\/b>: First \u2014 I agree, this is purely a Postgres story. I think if we have some kind of direct IO and a buffer pool almost for all memory, such an approach won't work \u2014 the plans will be different. But for now, we are only working with Postgres and are not thinking about the others.<\/i><\/p>\n

Regarding Kubernetes. You keep telling everyone that we have a persistent database. If an instance crashes, the main thing is to save the disk. So we also have the entire platform in Kubernetes, with the Postgres component separated (although it will be there at some point). So it's like this: the instance crashed, but we saved its PV and simply connected it to another (new) instance, as if nothing had happened.<\/i><\/p>\n

DS<\/b>: From my point of view, we create pods in Kubernetes. K8s is elastic: nodes are ordered as needed. The task is simply to create a pod and specify that it needs X resources, and then K8s will handle the rest. However, storage support in Kubernetes is still unstable: in 1.16<\/a><\/noindex>, in 1.17<\/a><\/noindex> (this release came out weeks<\/i> ago) these features are still in beta.<\/p>\n

It will take about six months to a year for it to stabilize, or at least to be declared as such. At that point, the ability for snapshots and resizing will fully solve your task. Because you have a base. Yes, it might not be very fast, but speed depends on what\u2019s \u2018under the hood\u2019, as some implementations can copy and use copy-on-write at the disk subsystem level.<\/p>\n

NS<\/b>: Here, it also needs all the engines (Amazon, Google...) to start supporting this version \u2014 that also takes some time.<\/i><\/p>\n

DS<\/b>: For now, we are not using them. We are using our own.<\/p>\n

Local development under Kubernetes.<\/h2>\n

\nNS<\/b>: Have you ever encountered the need to spin up all pods on a single machine and perform a quick test? To quickly obtain a proof of concept and check that the application works in Kubernetes without dedicating a whole bunch of machines to this task? There\u2019s Minikube, right?<\/i><\/p>\n

DS<\/b>: It seems to me that this case of deploying on a single node is solely about local development. Or some manifestations of such a pattern. There\u2019s also Minikube<\/a><\/noindex>, there\u2019s k3s<\/a><\/noindex>, KIND<\/a><\/noindex>. We're moving towards using Kubernetes IN Docker. We\u2019ve just started working with it for tests.<\/p>\n

NS<\/b>: I used to think this was an attempt to wrap all pods into a single Docker image. But it turned out to be something completely different. There are still separate containers, separate pods \u2014 just in Docker.<\/i><\/p>\n

DS<\/b>: Yes. And they\u2019ve made a rather amusing simulation, but the point is... We have a deployment utility \u2014 werf<\/a><\/noindex>. We want to create a mode in it \u2014 conditionally werf up<\/code>: \u201cSet up my local Kubernetes.\u201d And then launch a conditional werf follow<\/code>. Then the developer can edit in the IDE while a process is running in the system that sees changes, rebuilds images, and redeploys them in the local K8s. This is how we want to tackle the problem of local development.<\/p>\n

Snapshots and cloning databases in the realm of K8s<\/h2>\n

\nNS<\/b>: Returning to copy-on-write. I noticed that clouds also have snapshots. They work differently. For example, in GCP: you have a multi-terabyte instance on the US East Coast. You periodically create snapshots. You bring up a disk copy from a snapshot on the West Coast \u2014 within a few minutes, everything is ready, it works very quickly, you just need to fill the cache in memory. But these clones (snapshots) are meant to \u2018provision\u2019 a new volume. This is great when you need to create multiple instances.<\/i><\/p>\n

But for testing, I think the snapshots you\u2019re talking about in Docker or the ones I describe in ZFS, btrfs, and even LVM\u2026 \u2014 they allow you not to generate actual new data on a single machine. In the cloud, you will have to pay for them each time and wait not seconds but minutes (and in the case of lazy loading<\/a><\/noindex>, possibly even hours).<\/i><\/p>\n

Instead, you can get this data in a second or two, run the test, and discard it. These snapshots solve different tasks. In the first case \u2014 to scale up and obtain new replicas, and in the second \u2014 for testing.<\/i><\/p>\n

DS<\/b>: I disagree. Properly implementing volume cloning is a cloud task. I haven't looked at their implementation, but I know how we do it on hardware. We have Ceph, where you can say to any physical volume (RBD<\/a><\/noindex>) clone<\/i> and get a second volume with the same characteristics in just tens of milliseconds, IOPS<\/a><\/noindex>and so on. It\u2019s important to understand that there\u2019s a clever copy-on-write mechanism inside. Why doesn\u2019t the cloud do the same? I\u2019m sure they are trying to implement it one way or another.<\/p>\n

NS<\/b>: But they will still take seconds, tens of seconds, to spin up an instance, set up Docker, etc.<\/i><\/p>\n

DS<\/b>: Why do we need to spin up an entire instance? We have instances with 32 cores, 16 cores\u2026 and they can handle a certain number of workloads\u2014say, four. When we request a fifth, an instance will start, and then it will be deleted.<\/p>\n

NS<\/b>: Yes, interestingly, in Kubernetes, there\u2019s a different narrative. Our database is not in K8s, and it's a single instance. However, cloning a multi-terabyte database takes no more than two seconds.<\/i><\/p>\n

DS<\/b>: That's cool. But my initial point is that this is not a generic solution. Yes, it's great, but it only works for Postgres and only on one node.<\/p>\n

NS<\/b>: It's not just for Postgres: these plans, as I described, will only work that way for it. But if we don't worry about plans, and we just need all the data for functional testing, then it can work with any DBMS.<\/i><\/p>\n

DS<\/b>: Many years ago, we did something similar with LVM snapshots. That's classic. That approach was very actively used. The downside of stateful nodes \u2014 it's a hassle. Because you can't let them drop, you always have to keep track of them\u2026<\/p>\n

NS<\/b>: Do you see any possibility of a hybrid here? Suppose a stateful pod is running for several users (many testers). We have one volume, but thanks to the file system, the clones are local. If a pod crashes, the disk remains \u2014 the pod will restart, read all the cloning information, bring everything back, and say: 'Here are your clones running on these ports, continue working with them.'<\/i><\/p>\n

DS<\/b>: Technically, this means that within Kubernetes this is one pod, inside which we run multiple Postgres instances.<\/p>\n

NS<\/b>: Yes. It has a limit: for instance, no more than 10 people can work with it at once. If 20 are needed, we will spin up a second such pod. It\u2019s completely feasible to clone it, resulting in a second full volume with the same 10 'thin' clones. Do you not see that possibility?<\/i><\/p>\n

DS<\/b>Security questions need to be added here. This arrangement implies that this pod has elevated privileges (capabilities) because it can perform non-standard operations on the file system... But I reiterate: I believe that in the medium term, Kubernetes will fix storage issues, and the cloud providers will resolve all the problems with volumes \u2013 everything will just \"work.\" There will be resizing, cloning\u2026 We have a volume - we say: \"Create a new one based on this,\" and after a second and a half we get what we need.<\/p>\n

NS<\/b>: I don't believe in a second and a half for many terabytes. With Ceph, you do it yourself, but you're talking about cloud services. Go to the cloud, create a clone of an EBS volume of many terabytes on EC2, and see what kind of performance you'll get. It won't take a few seconds. I'm very interested in when they'll reach that level. I understand what you're talking about, but I allow myself to disagree.<\/i><\/p>\n

DS<\/b>: Okay, but I said in the medium term, not short term. Within a few years.<\/p>\n

About the PostgreSQL operator from Zalando<\/h2>\n

\nDuring the middle of this meeting, Alexey Klyukin, a former developer from Zalando, also joined in to tell the story of the PostgreSQL operator:<\/p>\n

It's great that this topic is even being touched on: both Postgres and Kubernetes. When we started working on it at Zalando in 2017, it was a topic that everyone wanted to engage with, but no one was doing it. Everyone was already using Kubernetes, but when they asked how to deal with databases, even people like Kelsey Hightower<\/a><\/noindex>, who preached K8s, said something like this:<\/p>\n

\"Go for managed services and use them, don't run databases in Kubernetes. Otherwise, your K8s might decide to upgrade, take down all nodes, and your data will disappear far, far away.\"<\/i><\/p>\n

We decided to create an operator that would, against this advice, run Postgres databases in Kubernetes. And we had a good foundation \u2014 Patroni<\/a><\/noindex>This is automatic failover for PostgreSQL, implemented correctly, i.e., using etcd, Consul, or ZooKeeper as the cluster information storage. A storage that will provide consistent information to everyone who asks, for example, what the current leader is \u2013 despite everything being distributed \u2013 to avoid split brain issues. Plus, we had Docker image<\/a><\/noindex> for it.<\/p>\n

Overall, the need for auto failover arose for the company after migrating from an internal hardware data center to the cloud. The cloud was based on a homegrown PaaS (Platform-as-a-Service) solution. It's open-source, but setting it up required considerable effort. It was called STUPS<\/a><\/noindex>.<\/p>\n

Initially, there was no Kubernetes. More precisely, when the homegrown solution was deployed, K8s already existed, but it was so immature that it wasn't suitable for production. This was around 2015 or 2016. By 2017, Kubernetes had matured enough to necessitate migration.<\/p>\n

We already had a Docker container. There was a PaaS that used Docker. Why not try K8s? Why not write our own operator? Murat Kabilov, who joined us from Avito, started this as a project on his own initiative \u2014 just to 'play around,' \u2014 and the project 'took off.'<\/p>\n

But overall, I wanted to talk about AWS. Why there was historically code related to AWS\u2026<\/p>\n

When you launch anything in Kubernetes, you need to understand that K8s is a work in progress. It is constantly evolving, improving, and sometimes even breaking. You have to pay close attention to all the changes in Kubernetes, be prepared to dive into it if necessary and understand how it works in detail \u2014 possibly more than you'd like. This is the case with any platform where you run your databases...<\/p>\n

So, when we created the operator, we had Postgres working with an external volume (in this case, EBS, since we were working on AWS). The database grew, and at some point, a resize became necessary: for example, the original EBS size was 100 TB, the database grew to that size, and now we want to make the EBS 200 TB. How? You could dump\/restore to a new instance, but that's slow and involves downtime.<\/p>\n

Therefore, we desired a resize operation that would increase the EBS volume and then instruct the file system to utilize the new space. We accomplished this, but at that time, Kubernetes had no API for resize operations. Since we were working on AWS, we wrote code for its API.<\/p>\n

There's nothing stopping you from doing the same for other platforms. The operator doesn't have a dependency that it can only run on AWS; it will work on other platforms as well. In general, this is an open source project: if someone wants to speed up the adoption of the new API \u2014 you're welcome to do so. There are GitHub<\/a><\/noindex>, pull requests \u2014 the Zalando team tries to respond to them promptly and promote the operator. As far as I know, the project participated<\/a><\/noindex> in Google Summer of Code and some other similar initiatives. Zalando is actively working on it.\n<\/p><\/blockquote>\n

<\/p>\n

P.S. Bonus!<\/h2>\n

\nIf you are interested in PostgreSQL and Kubernetes, we also want to draw your attention to the fact that last week the next Postgres Tuesday took place, where Nikolai spoke with Alexander Kukushkin from Zalando<\/b>. The video from it is available here<\/a><\/noindex>.<\/p>\n

P.P.S.<\/h2>\n

\nAlso read in our blog:<\/p>\n