Yandex's contribution to the following databases will be considered.
- clickhouse
- Odyssey
- Recovery to a Point in Time (WAL-G)
- PostgreSQL (including logerrors, Amcheck, heapcheck)
- Greenplum
Video:
hello world! My name is Andrey Borodin. And in Yandex.Cloud, I am developing open relational databases for the benefit of Yandex.Cloud and Yandex.Cloud clients.
In this talk, we'll talk about the challenges of open databases at scale. Why is it important? Because little, little problems, which, like mosquitoes, then become elephants. They get big when you have a lot of clusters.
But that's not the point. Incredible things happen. Things that happen in one in a million cases. And in a cloud environment, you have to be prepared for this, because incredible things become very likely when something exists at scale.
But! What are the advantages of open databases? The fact that you have a theoretical opportunity to deal with any problem. You have the source code, you have programming knowledge. We combine and work.
What approaches are there in working on open source software?
- The most understandable approach is to use software. If you use protocols, if you use standards, if you use formats, if you write requests on open source software, then you already support it.
- You make its ecosystem bigger. You make it more likely to find a bug early. You increase the reliability of this system. You increase the availability of developers in the market. You improve this software. You are already a contributor if you just did up getting style and tinkered with it.
- Another understandable approach is to sponsor open source software. For example, the well-known Google Summer of Code program, when Google pays a large number of students from all over the world understandable money to develop open source software projects that meet certain license requirements.
- This is a very interesting approach because it allows you to develop the software without shifting the focus away from the community. Google, as a technology giant, does not say that we want this feature, we want to fix this bug, and that's where we need to dig. Google says, “Do what you do. Just keep working the way you've been working and everything will be fine."
- The next approach to participation in open source is participation. When you have a problem in open source software and there are developers, then your developers start solving problems. They begin to make your infrastructure more efficient, your programs faster and more reliable.
One of the most famous Yandex open source projects is ClickHouse. This is a database that was born as a response to the challenges facing Yandex.Metrica.
And as a database, it was made in open source in order to create an ecosystem and develop it together with other developers (not only within Yandex). And now it is a big project in which many different companies are involved.
In Yandex.Cloud, we made ClickHouse on top of Yandex Object Storage, i.e. on top of cloud storage.
Why is it important in the cloud? Because any database works in this triangle, in this pyramid, in this hierarchy of memory types. You have fast but small registers and cheap large but slow SSDs, HDDs and other block devices. And if you are efficient at the top of the pyramid, then you have a fast database. if you are effective at the bottom of this pyramid, then you have a scaled database. And in this regard, adding another layer from the bottom is a logical approach to increase the scalability of the database.
How could it be done? This is an important point in this report.
- We could implement ClickHouse over MDS. MDS is an internal Yandex cloud storage interface. It is more complex than the common S3 protocol, but it is more suitable for a block device. It is better suited for writing data. It requires more programming. Programmers will program, it's even good, interesting.
- S3 is a more common approach that makes for a simpler interface at the cost of less adaptation to certain types of workloads.
Naturally, wanting to provide functionality to the entire ClickHouse ecosystem and perform the task that is needed inside Yandex.Cloud, we decided to make it so that the entire ClickHouse community would benefit from it. We implemented ClickHouse over S3, not ClickHouse over MDS. And that's a lot of work.
Links:
"Filesystem abstraction layer"
AWS SDK S3 integration
"Base implementation of IDisk interface for S3"
"Integration of log storage engines with IDisk interface"
"Log engine support for S3 and SeekableReadBuffer"
"Storage Stripe Log S3 support"
"Storage MergeTree initial support for S3"
MergeTree full support for S3
"Support ReplicatedMergeTree over S3"
"Add default credentials and custom headers for s3 storage"
"S3 with dynamic proxy configuration"
S3 with proxy resolver
This is a pull request list for ClickHouse's virtual file system implementation. That's a lot of pull requests.
Links:
"DiskS3 hardlinks optimal implementation"
"S3 HTTP client - Avoid copying response stream into memory"
"Avoid copying entire response stream into memory in S3 HTTP
customer"
"Ability to cache mark and index files for S3 disk"
"Move parts from DiskLocal to DiskS3 in parallel"
But the work didn't end there. After the feature was made, it took some more work to optimize this functionality.
Links:
"Add SelectedRows and SelectedBytes events"
"Add profiling events from S3 request to system.events"
"Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds"
And then it was necessary to make it diagnosable, establish monitoring and make it manageable.
And all this was done so that the entire community, the entire ClickHouse ecosystem would receive the result of this work.
Let's move on to transactional databases, to OLTP databases, which are closer to me personally.
This is the open source DBMS development division. These guys are doing street magic to improve transactional open databases.
One of the projects that we can use as an example to talk about how and what we do is the Connection Pooler in Postgres.
Postgres is a process database. This means that the database should include as few network connections as possible that work with transactions.
On the other hand, in a cloud environment, a typical situation is when a thousand connections come to one cluster at once. And the task of the connection pooler is to pack a thousand connections into a small number of server connections.
We can say that the connection pooler is a telephone operator who shifts the bytes so that they effectively reach the database.
Unfortunately, there is no good Russian word for a connection pooler. It is sometimes referred to as multiplexer connections. If you know what to call the connection pooler, then be sure to tell me, I will be very happy to speak the correct Russian technical language.
We explored connection poolers that were suitable for a managed postgres cluster. And PgBouncer was the best fit for us. But we've run into a number of issues with PgBouncer. Many years ago, Volodya Borodin made reports that we use PgBouncer, we like everything, but there are nuances, there is something to work on.
And we worked. We fixed the problems that we encountered, we patched Bouncer, we tried to attribute the pull request to the upstream. But fundamental single-threading was hard to work with.
We had to cascade from patched Bouncers. When we have a lot of single-threaded Bouncers, then on the top layer, connections are transferred to the inner layer of Bouncers. It's a poorly managed, hard to build and hard to scale back and forth system.
We have come to the conclusion that we have created our own connection pooler, which is called Odyssey. We wrote it from scratch.
In 2019, at PgCon, I presented this puller to the developer community. Now we have a little less than 2 stars on GitHub, that is, the project is alive, the project is popular.
And if you create a Postgres cluster in Yandex.Cloud, then it will be a cluster with built-in Odyssey, which is reconfigured when the cluster is scaled back and forth.
What have we learned from this project? Launching a competing project is always an aggressive step, it is a last resort when we say that there are problems that are not solved quickly enough, not solved in the time intervals that would suit us. But it is an effective measure.
PgBouncer started to develop faster.
And now there are other projects. For example, pgagroal, which is developed by Red Hat developers. They pursue similar goals and implement similar ideas, but, of course, with their own specifics, which are closer to pgagroal developers.
Another case of working with the postgres community is restoring to a point in time. This is a disaster recovery, this is a restore from a backup.
There are many backups and they are all different. Almost every Postgres vendor has their own backup solution.
If we take all the backup systems, make a matrix of features and jokingly calculate the determinant in this matrix, then it will be zero. What does this mean? What if you take a particular backup, then it cannot be assembled from the pieces of all the others. It is unique in its implementation, it is unique in its purpose, it is unique in the ideas that are embedded in it. And they are all specific.
While we were looking into this issue, CitusData launched the WAL-G project. This is a backup system that was made with an eye on the cloud environment. CitusData is now part of Microsoft. And at that moment, we really liked the ideas that were laid down on the initial releases of WAL-G. And we started to contribute to this project.
Now there are many dozens of developers in this project, but the top 10 WAL-G contributors include 6 Yandexoids. We brought a lot of our ideas there. And, of course, we implemented them ourselves, tested them ourselves, rolled them out into production ourselves, use them ourselves, figure out where to go next, while interacting with the large WAL-G community.
And from our point of view, now this backup system, including taking into account our efforts, has become optimal for the cloud environment. This is better than backing up Postgres to the cloud.
What does it mean? We were pushing a pretty big idea: backups should be secure, cheap to run, and as fast as possible to restore.
Why should it be cheap to operate? When nothing is broken, you don't have to know that you have backups. Everything works fine, you waste as little CPU as possible, you use as little of your disk resources as possible, and you send as few bytes as possible to the network so as not to interfere with the payload of your valuable services.
And when everything broke down, for example, the admin dropped the data, something went wrong, and you urgently need to return to the past, you recover with all the money, because you want your data back quickly and intact.
And we have advanced this simple idea. And, as it seems to us, we managed to realize it.
But that's not all. We wanted one more little thing. We wanted many different databases. Not all of our clients use Postgres. Some use MySQL, MongoDB. In the community, other developers have supported FoundationDB. And this list is constantly expanding.
The community loves the idea of having a database run in a managed environment in the cloud. And developers maintain their databases, which can be copied uniformly along with Postgres by our backup system.
What have we learned from this story? Our product, as a development division, is not lines of code, these are not operators, these are not files. Our product is not pull requests. These are the ideas that we broadcast to the community. This is technological expertise and the movement of technology towards the cloud environment.
There is such a database as Postgres. I like the Postgres core the best. I spend a lot of time developing the Postgres core with the community.
But here I must say that Yandex.Cloud has an internal installation of managed databases. And it started a long time ago in Yandex.Mail. The kind of expertise that now led to managed Postgres was accumulated when the mail wanted to drop into Postgres.
Mail has very similar requirements to the cloud. It needs you to be able to scale for unexpected exponential growth at any point in your data. And the mail already had a load with some hundreds of millions of mailboxes of a huge number of users who constantly make many requests.
And it was quite a big challenge for the Postgres development team. Then all the problems we encountered were reported to the community. And these problems were corrected, and were corrected by the community in places even at the level of paid support for some other databases and even better. That is, you can send a letter to PgSQL hacker and get a response within 40 minutes. Paid support in some databases may think that there are more priority things than your bug.
Now an internal Postgres installation is some petabytes of data. These are some millions of requests per second. These are thousands of clusters. She is very large.
But there is a nuance. She lives not on fashionable network drives, but on fairly simple hardware. And there is a test environment specifically for interesting new things.
And at some point in the test environment, we received such a message, which indicates that the internal invariants of database indexes were violated.
An invariant is some kind of relation that we expect to always hold.
A very critical situation for us. It indicates that some data may have been lost. And data loss is something downright catastrophic.
The general idea that we follow in managed databases is that even with the effort, it will be difficult to lose data. Even if you intentionally remove them, you still need to ignore their absence for a long period of time. Data integrity is a religion we follow quite diligently.
And then a situation arises that suggests that there may be a situation for which we may not be ready. And we began to prepare for this situation.
The first thing we did was bury the logs from these thousands of clusters. We found which of the clusters are located on disks with problematic firmware that lost updating data pages. Marked up all Postgres code data. And those messages that indicate violations of internal invariants, we have marked with a code that is designed to detect data corruption.
This patch was practically accepted by the community without much discussion, because in each case it was obvious that something bad had happened, it should be reported in the log about it.
After that, we came to the conclusion that we have monitoring that scans the logs. And in case of suspicious messages, he wakes up the attendant, and the attendant fixes it.
But! Scanning logs is a cheap operation on one cluster and catastrophically expensive for thousands of clusters.
We have written an extension called . It creates a view of the database in which past error statistics can be cheaply and quickly populated. And if you need to wake up the attendant, then we will find out about it not by scanning gigabyte files, but by extracting a few bytes from the hash table.
This extension has been accepted, for example, in the repository for . If you want to use it, you can install it yourself. Of course it's open source.
But that's not all. We started using Amcheck, a community-created extension for finding invariant violations in indexes.
And we found out that if you exploit it on a scale, then there are bugs. We started making them. Our corrections have been accepted.
We found that this extension is unable to parse GiST & GIT indexes. We made them support. But this support is still being discussed by the community, because this is a relatively new functionality and there are a lot of details.
And we also found that in the case of checking indexes for violations on the replication leader, everything works well on the master, but on replicas, on the follower, the search for corruption is not so effective. Not all invariants are verified. And one invariant bothered us very much. And we talked with the community for a year and a half in order to enable this check on replicas.
We wrote code that should follow all can ... protocols. Had a long discussion with Peter Gagan of Crunchy Data about this patch. He had to slightly modify the existing B-tree in Postgres in order to accept this patch. He was accepted. And now checking indexes on replicas has also become quite effective in order to detect the violations that we encountered. That is, these are the violations that can be caused by errors in disk firmware, bugs in Postgres, bugs in the Linux kernel, hardware problems. Quite an extensive list of sources of problems for which we were preparing.
But besides indexes, there is such a part as heap, i.e. the place where the data is stored. And there are not many invariants that could be checked.
We have an extension called Heapcheck. We started developing it. And in parallel with us, the EnterpriseDB company also began to write a module, which they called Heapcheck in the same way. Only we called it PgHeapcheck, and they just called it Heapcheck. They have it with similar functions, with a slightly different signature, but with the same ideas. They implemented them a little bit better in places. And previously laid out in open source.
And now we are in the business of developing their extension, because this is no longer their extension, but a community extension. And in the future, this is a part of the kernel that will be supplied to everyone in order to be able to know about future problems in advance.
In some places, we even came to the conclusion that we have false positives in our monitors. For example, system 1C. When using a database, Postgres sometimes writes data to it that it can read itself, but pg_dump cannot read.
This situation looked like corruption to our problem detection system. The attendant was awakened. The attendant looked to see what was going on. After a while, a client came and said that I had a problem. The attendant explained what the problem was. But the problem is in the Postgres kernel.
I found a discussion of this feature. And he wrote that we encountered this feature and it was unpleasant, a person woke up at night in order to figure out what it was.
The community responded: “Oh, really, it needs to be fixed.”
I have a simple analogy. If you walk in a shoe that has got a grain of sand, then, in principle, you can go further - that's okay. If you sell shoes to thousands of people, then let's make shoes without sand at all. And if one of the users of your shoes is going to run a marathon, then you want to make very good shoes, and then scale to all your users. And such unexpected users are always in the cloud environment. There are always users who operate the cluster in some original way. You must always prepare for this.
What have we learned here? We learned a simple thing: the most important thing is to explain to the community that there is a problem. If the community has recognized the problem, then there is a natural competition to solve the problem. Because everyone wants to solve an important problem. All vendors, all hackers understand that they themselves can step on this rake, so they want to eliminate them.
If you are working on some problem, but it does not bother anyone except you, but you are working on it systematically and it was eventually considered a problem, then your pull request will definitely be accepted. Your patch will be accepted, your improvements or even requests for improvements will be considered by the community. After all, we make the database better for each other.
An interesting database is Greenplum. It's a highly parallel database based on the Postgres codebase that I'm familiar with.
And Greenplum has an interesting functionality - it's append optimized tables. These are tables in which you can quickly add. They can be either columnar or inline.
But there was no clustering, that is, there was no functionality when you can order the data located in the table in accordance with the order that is in one of the indexes.
The guys from the taxi came to me and said: “Andrey, you know Postgres. And it's almost the same here. Switching for 20 minutes. Take it and do it." I thought, yes, I know Postgres, switching to 20 minutes - I need to do this.
But no, it wasn't 20 minutes, I've been writing this for months. At the PgConf.Russia conference, I approached Heikki Linakangas from Pivotal and asked: “Are there any problems with this? Why is there no clustering of append optimized tables?”. He says: “You take the data. Sorting, translating. It's just work." Me: “Oh, yes, you just have to go and do it.” He says, "Yes, we need free hands to do it." I thought that this is exactly what I need to do.
And a few months later I sent a pull request that implemented this functionality. This pull request was looked at in Pivotal in conjunction with the community. Of course, there were bugs.
But the most interesting thing is that when this pull request was merged, bugs were found in Greenplum itself. We have found that heap tables sometimes break transactionality when clustered. And that's the thing that needs to be fixed. And she's in the place I just touched. And my natural reaction was - well, let me do this too.
I fixed this bug. Sent a pull request to fixes. He was slain.
After that, it turned out that this functionality needs to be obtained in the Greenplum version for PostgreSQL 12. That is, the adventures for 20 minutes continue with new interesting adventures. It was interesting to touch the current development, where the community saws new and most important features. It's been dead.
But it didn't end there. After all, it turned out that it was necessary to write documentation for all this.
I started writing documentation. Luckily, the docs from Pivotal came along. For them, English is their native language. They helped me with the documentation. In fact, they themselves rewrote what I suggested into real English.
And that seemed to be the end of the adventure. And do you know what happened next? The guys from the taxi came to me and said: "There are still two adventures, each for 10 minutes." And what should I tell them? I said that now I will make a report on the scale, then we will see your adventures, because this is an interesting job.
What have we learned in this case? The fact that working with open source is always working with a specific person, it is always working with the community. Because at every single stage I worked with some developer, with some tester, with some hacker, with some documenter, with some architect. I didn't work with Greenplum, I worked with the people around Greenplum.
But! There is another important point - it's just work. That is, you come, drink coffee, write code. All sorts of simple invariants work. Do it right - it will be okay! And it's pretty interesting work. There is a request for this work from Yandex.Cloud clients, users of our clusters both inside Yandex and outside. And, I think that the number of projects in which we participate will increase and the depth of our involvement will also increase.
That's all. Let's move on to questions.
Question session
Hello! We have another Q&A session. And in the studio Andrey Borodin. This is the person who just told you about the contribution of Yandex.Cloud and Yandex to open source. Our report is not entirely about the Cloud, but at the same time, we are based on such technologies. If it wasn't for what you did inside Yandex, there was no service in Yandex.Cloud, so thank you from me personally. And the first question from the broadcast: "What is written in each of the projects that you mentioned?".
The backup system in WAL-G is written in Go. This is one of the newest projects we have worked on. He is literally only 3 years old. And the database is often about reliability. And this means that the databases are quite old and they are usually written in C. The Postgres project started about 30 years ago. Then the C89 was the right choice. And Postgres is written on it. More modern databases such as ClickHouse are already written in C++ usually. All system development is based around C and C++.
A question from our financial manager, who is in charge of spending in the Cloud: "Why does the Cloud spend money on supporting open source?".
There is a simple answer for the financial manager. We do this to improve our services. In what way can we do better? We can make things more efficient, faster, make things more scalable. But for us, this story is primarily about reliability. For example, in the backup system, we will review 100% of the patches that apply to it. We know what the code is. And we are more relaxed rolling out new versions to production. That is, first of all, it is about confidence, about readiness for development and about reliability
Another question: “Does the requirements of external users who live in Yandex.Cloud differ from those of internal users who live in the internal Cloud?”.
The load profile is, of course, different. But from my department's point of view, all special and interesting cases are created on a non-standard load. Developers with imagination, developers doing unexpected things, they are equally likely to be found both inside and outside. In this regard, we are all about the same. And, probably, the only important feature inside the Yandex operation of databases will be that inside Yandex we have a teaching. At some point, some access zone completely goes into the shadows, and all Yandex services must somehow continue to function, despite this. Here is a small difference. But it creates a lot of R&D at the interface between the database and the network stack. Otherwise, external and internal installations create the same feature requests and similar requests for improved reliability and performance.
The next question is: “How do you personally feel about the fact that much of what you do is used by other Clouds?”. We won't name specific ones, but many of the projects that we did in Yandex.Cloud are used in other people's clouds.
This is cool. First, it is a sign that we have done something right. And it scratches the ego. And we are more confident that we made the right decision. On the other hand, it is hope that in the future it will bring us new ideas, new requests from third-party users. Most issues on GitHub are created by individual sysadmins, individual DBAs, individual architects, individual engineers, but sometimes people with systematic experience come in and say that in 30% of certain cases we have such and such a problem and let's think about how to solve it. This is what we are looking forward to the most. We look forward to sharing experiences with other cloud platforms.
You talked a lot about the marathon. I know that you ran a marathon in Moscow. As a result? Overtook the guys from Postgres Pro?
No, Oleg Bartunov runs very fast. He finished an hour before me. Overall, I'm happy with what I ran. For me, just finishing was an achievement. In general, it's surprising that there are so many runners in the postgres community. It seems to me that there is some relationship between aerobic sports and the desire for system programming.
Are you saying there are no runners in ClickHouse?
I know for sure that they are there. ClickHouse is also a database. By the way, now Oleg writes to me: “Let's go for a run after the report?”. This is a great idea.
Another question from the broadcast from Nikita: “Why did you fix the bug in Greenplum yourself, and didn’t give it to juniors?” True, it is not very clear here what kind of bug and in what service, but, probably, the one you talked about is meant.
Yes, in principle, you could give it to someone. It was just the code that I just changed. And it was natural to continue to do it right there. In principle, the idea of sharing expertise with a team is a good idea. We will definitely share Greenplum tasks among all members of our division.
Since we are talking about juniors, such a question. The person decided to create the first commit in Postgres. What does he need to do to make the first commit?
This is an interesting question: "Where to start?". It's usually hard to start with something in the kernel. Postgres, for example, has a to-do list. But in fact, this is a list of what they tried to do, but did not work out. These are complex things. And you can usually find some utilities in the ecosystem, some extensions that can be improved, that get less attention from the kernel developers. And, accordingly, there are more points for growth. At the Google Summer of code program every year, the postgres community puts up many different topics that could be addressed. This year we seem to have had three students. One even wrote in WAL-G on topics that are important for Yandex. Things are easier in Greenplum than in the Postgres community because the Greenplum hackers are very good with pull requests and start reviewing right away. Sending a patch to Postgres is some kind of story for months, and Greenplum will come in a day and see what you have done. Another thing is that Greenplum needs to solve actual problems. Greenplum is not widely exploited, so finding your problem is quite difficult. And first of all it is necessary to solve, of course, problems.
Source: habr.com