I want to share the transcript of Georgy Rylov's report from early 2020 titled "WAL-G: new opportunities and community expansion".
Open-source maintainers face many challenges as they grow. How do you write more required features, fix more issues, and keep up with more pull requests? Using WAL-G (a backup tool for PostgreSQL) as an example, I will discuss how we addressed these challenges by launching an Open-source development course at the university, what we achieved, and where we plan to go next.
Hello again everyone! I am a developer at Yandex from Yekaterinburg. Today, I will talk about WAL-G.
The title of the report didn't specify that this was about backups. Does anyone not know what WAL-G is? Or does everyone know? Raise your hand if you don't know. Wow, you came to the report and don't know what it's about.
Let me explain what will be discussed today. It so happens that our team has been involved in backups for quite some time. This is yet another talk in a series where we share how we store data safely, reliably, conveniently, and efficiently.
In previous sessions, there have been many talks by Andrey Borodin and Vladimir Leskov. There were many of us. And we have been discussing WAL-G for many years.
clck.ru/F8ioz —
clck.ru/Ln8Qw —
This talk will differ a bit from the others in that those focused more on the technical aspect, while here I will discuss how we faced challenges associated with community growth. And how we came up with a small idea that helps us manage that.
A few years ago, WAL-G was a fairly small project that we inherited from Citus Data. We just took it over. It was developed by one person.
And there was only one thing that WAL-G lacked:
- Backup from a replica.
- There were no incremental backups.
- There were no WAL-Delta backups.
- And a lot of other things were missing.
Over these few years, WAL-G has grown significantly.
By 2020, all of the above had been introduced. Additionally, we now have:
- Over 1,000 stars on GitHub.
- 150 forks.
- About 15 open PRs.
- And many contributors.
- And open issues constantly. This is despite the fact that we literally check in every day and do something about it.
We concluded that this project requires more of our attention, even when we ourselves don't need to implement anything for our Managed Databases service at Yandex.
And somewhere in the autumn of 2018, an idea came to us. Normally, the team has a few ways to develop features or fix bugs when you lack manpower. For instance, you could hire another developer and pay them. Or you might bring in an intern for a while and pay them a salary too. But there’s also a significant group of people, some of whom already know how to write code. It's just that you don’t always know the quality of that code.
We thought about it and decided to try attracting students. However, students won’t be involved in everything. They will do only a part of the work. For example, they might write tests, fix bugs, or implement features that don’t affect the core functionality. Core functionality includes backup creation and recovery. If a bug occurs in backup creation, we risk data loss. And nobody wants that, of course. Everyone wants reliability. Therefore, we don't want to allow code we trust less than our own. In other words, any non-critical code is what we would like to receive from our additional hands.
Under what conditions is a student’s PR accepted?
- They are required to cover their code with tests. Everything must pass in CI.
- Additionally, we go through two reviews. One by Andrey Borodin and one by me.
- Moreover, to ensure it won't break anything in our service, I separately deploy the build with that commit. We verify with end-to-end tests that nothing is failing.
Special course on Open Source
A little about why this is needed and why I think it's a cool idea.
The benefit for us is obvious:
- We get extra hands.
- And we look for team candidates among capable students who write solid code.
What’s the benefit for students?
They may be less obvious, as students, at least, don’t get paid for the code they write; they only receive grades.
I asked them about it. And from their words:
- Experience as an Open Source contributor.
- To get a line in their CV.
- To showcase themselves and go through an interview at Yandex.
- To become a participant in GSoC.
- +1 special course for those who want to write code.
I won't go into how the course was structured. I will only say that WAL-G was the main project. Additionally, we included projects like Odyssey, PostgreSQL, and ClickHouse in this course.
And we provided tasks not only in this course but also issued diplomas and coursework.
And what about the benefit for users?
Now let's move on to the part that concerns you more. What is the benefit to you? The benefit is that students fixed many bugs and made feature requests that you asked us to implement.
And let me tell you about things that you have been wanting for a long time and which have been realized.
Support for tablespaces. Support for tablespaces in WAL-G has probably been expected since the release of WAL-G, because WAL-G is a successor to another backup tool, WAL-E, which supported database backups with tablespaces.
Let me briefly remind you what this is and why it is needed. Typically, all your Postgres data resides in a single directory on the file system called the base directory. And this directory already contains all the files and subdirectories required by Postgres.
Tablespaces are directories that contain Postgres data, but they do not reside outside the base directory. The slide shows that tablespaces are outside the base directory.
How does this look for Postgres? In the base directory, there is a separate subdirectory called pg_tblspc. Inside it are symlinks to the directories where Postgres data actually resides outside the base directory.
When you use all this, these commands may look something like this for you. That is, you create a table in a specified tablespace and check where it currently resides. The last two lines, the two recently executed commands, indicate that there is a certain path. But in reality, this is not the actual path. This is a path with a prefix from the base directory to the tablespace. From there, it is matched with a symlink that points to your actual data.
We don't use all of this in our team, but many other WAL-E users used it and wrote to us that they wanted to switch to WAL-G, but this hindered them. Now it is supported.
Another feature that our special course brought us is catchup. People who have worked more with Oracle than with Postgres probably know about catchup.
A brief overview of what this is. This is how a cluster topology might typically look in our service. We have a master and a replica that streams the write-ahead log from it. The replica indicates to the master which LSN it is currently at. Simultaneously, the log might be archived. In addition to archiving the log, backups are also sent to the cloud, along with delta backups.
What could be the problem? When you have a fairly large database, it can happen that your replica starts to lag significantly behind the master. It lags so much that it can never catch up. This problem usually needs to be addressed somehow.
The simplest way to resolve this is to remove the replica and reinitialize it since it will never catch up, and we need to deal with the problem. However, this process is quite lengthy because restoring a full backup of a 10 TB database is very time-consuming. We would like to do this as quickly as possible when such issues arise. This is precisely what catchup is designed for.
Catchup allows you to use delta backups that are stored in the cloud in this way. You specify the LSN of the lagging replica and provide it in the catchup command to create a delta backup between that LSN and the LSN where your cluster currently is. After that, you restore this backup to the replica that is lagging.
Other databases
Additionally, our students brought us a number of features. Since we at Yandex work with not only Postgres, but also MySQL, MongoDB, Redis, and ClickHouse, at one point we needed to ensure we could perform backups with point-in-time recovery for MySQL, and have the capability to upload them to the cloud.
We wanted to do this in a way similar to how WAL-G operates. We decided to experiment and see how it would all turn out.
Initially, without separating this logic, we wrote the code in a fork. We saw that we had some working model, and it was feasible. Later, we thought that our main community consists of Postgres users who utilize WAL-G. Therefore, we needed to separate these components. That is to say, when we modify the code for Postgres, we do not break MySQL, and vice versa.
The initial idea for separating this was to use the same approach as that used in PostgreSQL extensions. Essentially, to back up MySQL, you had to install some dynamic library.
However, the asymmetry of this approach is immediately evident. When backing up Postgres, you simply install a standard backup tool for Postgres and everything works fine. But for MySQL, it turns out that you install a backup tool for Postgres and also add a dynamic library for MySQL. That sounds a bit strange. We thought so too and decided that this is not the solution we need.
Different builds for Postgres, MySQL, MongoDB, Redis
But this allowed us, as we believe, to arrive at the right solution – to allocate different builds for different databases. This enabled us to isolate the logic tied to backups of various databases, which would interface with a common API implemented by WAL-G.
This is the part we wrote ourselves – before giving students the tasks. That is, this is precisely where they could potentially make mistakes, so we decided it was better to handle it ourselves to ensure everything would be done correctly.
After that, we issued the tasks. They were immediately taken up. Students were required to support three databases.
This is MySQL, which we have been backing up with WAL-G in this way for over a year now.
And now MongoDB is nearing production, where it is being fine-tuned. Essentially, we wrote the framework for all this. Then the students created some working components, and subsequently, we bring them to a state that we can accept in our production.
These tasks did not imply that students had to write complete backup tools for each of these databases. We were not faced with such a problem. Our issue was that we wanted point-in-time recovery and we wanted to back up to the cloud. We asked the students to write some code that would solve this. The students utilized already existing backup tools that somehow create backups, and then they integrated all this with WAL-G, which redirected everything to the cloud. They also added point-in-time recovery to this.
What else did the students contribute? They brought support for Libsodium encryption to WAL-G.
We also established backup retention policies. Now backups can be marked as permanent. This makes it more convenient for your service to automate the process of their storage.
What were the results of this experiment?
Initially, over 100 people registered for the course. I didn't mention at first that the university in Yekaterinburg is the Ural Federal University. That's where we made all the announcements. 100 people registered. However, much fewer actually started working, around 30 people.
Even fewer people completed the course, as it required writing tests for existing codes. They also needed to fix some bugs or implement certain features. A portion of the students still managed to complete the course.
Currently, students have fixed about 14 issues for this course and created 10 features of various sizes. It seems to me that this effectively replaces one or two developers.
Additionally, we issued diplomas and course projects. 12 students received diplomas, 6 of them have already defended their projects with top marks. The others haven't defended yet, but I believe they will do well too.
Plans for the future
What are our future plans?
At the very least, we want to implement the feature requests we've already heard from users. These include:
- Monitoring the accuracy of the timeline tracking in the HA-cluster backup archive. This can be done using WAL-G. I believe we can find students willing to take on this task.
- We already have a person responsible for transferring backups and WAL between clouds.
- We recently published an idea that we can further accelerate WAL-G by unpacking incremental backups without overwriting pages and optimizing the archives we send there.
Feel free to share your thoughts here
What was the purpose of this report? It’s to highlight that now, besides the four of us maintaining this project, we have additional hands that are quite numerous, especially if you message them directly. If you are backing up your data using WAL-G, or would like to switch to it, we can easily accommodate your wishes.
Here is the QR code and link. You can use them to share all your requests. For example, if there’s a bug we haven’t fixed, or a feature you really want that isn’t available in any of the backup solutions, including ours. Please make sure to mention it.
Questions
Hello! Thank you for the presentation! I have a question about WAL-G, but not about Postgres. WAL-G backs up MySQL and triggers an extra backup. If we take modern installations on CentOS and if you do yum install MySQL, it will install MariaDB. Since version 10.3, extra backup is not supported; only MariaDB backup is supported. How are you handling this?
Currently, we haven't tried backing up MariaDB. We had requests for FoundationDB support, but in general, if there is such a request, we can find people to do it. It’s not as time-consuming and complicated as I thought it would be.
Good day! Thank you for the presentation! I have a question about potentially new features. Are you ready to make WAL-G work with tapes to enable tape backups?
Are you referring to backup on tape storage?
Yes.
There is Andrey Borodin there, who can answer this question better than I can.
(Andrey) Yes, thanks for the question! We had a request to transfer backups to tape from cloud storage. And for this, the transfer between clouds. Because transferring between clouds is a generalized version of tape transfer. Additionally, we have a scalable architecture in terms of storage. By the way, many storages were written by students. And if you write a storage for tape, it will definitely be supported. We are ready to consider pull requests. You need to write a file and read a file. If you implement these in Go, it usually takes about 50 lines of code. Then tape will be supported in WAL-G.
Thank you for the presentation! An interesting development process. Backup is a significant part of functionality that should be well covered by tests. When you implemented functionality for new databases, were tests written by students as well, or did you write the tests yourself and then hand the implementation over to the students?
Tests were also written by students. But the students wrote more for features like new databases. They wrote integration tests and unit tests. If integration tests pass, i.e., at the moment — this is a scenario that you execute manually or that cron does for you, for example. So the scenario is quite clear.
Students do not have much experience. How much time does it take for reviews?
Yes, reviewing takes a significant amount of time. That is, usually when several committers come in and say, 'I did this, I did that', one needs to think and allocate at least half a day to figure out what they have written. Because the code needs to be read carefully. They didn't go through an interview. We don't know them very well, so this takes substantial time.
Thank you for the presentation! Earlier, Andrey Borodin stated that the archive_command in WAL-G should be called directly. But in the case of some patron cluster, we need additional logic to determine the node from which to send the wals. How do you solve this problem on your end?
What issue are you facing here? For example, do you have a synchronous replica from which you are taking a backup? Or what?
(Andrey) The fact is that WAL-G is indeed intended to be used without wrapping it in shell scripts. If something is missing, then let's write the logic that should be inside WAL-G. Regarding where the archiving should come from, we believe that archiving should be from the current master in the cluster. Archiving from a replica is a bad idea. There can be various scenarios with problems. In particular, issues with archiving timelines and other additional information. Thanks for the question!
(Clarification: We dropped the wrapping in shell scripts. )
Good evening! Thank you for the presentation! I was intrigued by the catchup feature you described. Have you ever encountered a situation where a replica fell behind and could not catch up? I did not find a description of this feature in the WAL-G documentation.
Catchup appeared literally in the 20s of January 2020. Perhaps it would be worth working on the documentation a bit more. We ourselves are writing it and not necessarily doing it perfectly. And we might need to start requiring students to write it as well.
Is it already released?
The pull request has been merged, that is, I checked it. I tried it on a test cluster. So far, we haven't had a situation where we could test this in a production example.
When to expect it?
I don’t know. Wait a month, we’ll definitely check.
Source: habr.com
