{"id":78887,"date":"2020-04-22T19:42:51","date_gmt":"2020-04-22T17:42:51","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov"},"modified":"2020-04-22T19:42:51","modified_gmt":"2020-04-22T17:42:51","slug":"wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov","title":{"rendered":"WAL-G: new capabilities and community expansion. Georgiy Rylov","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><strong>I would like to share the transcript of Georgy Rylov's report from early 2020 titled \"WAL-G: New Opportunities and Community Expansion\".<\/strong><\/p>\n<p><\/p>\n<p>Open-source maintainers face numerous challenges as they grow. How do they manage to write more and more required features, fix an increasing number of issues, and keep up with a growing number of pull requests? Using WAL-G (a backup tool for PostgreSQL) as an example, I will discuss how we addressed these challenges by launching a course on Open-source development at the university, what we achieved, and our future direction.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/c76e27949b782480284ab52791fdfe15.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Hello again everyone! I am a developer at Yandex from Yekaterinburg. Today, I will talk about WAL-G.<\/p>\n<p><\/p>\n<p>The title of the report didn't specify that this was about backups. Does anyone not know what WAL-G is? Or does everyone know? Raise your hand if you don't know. Wow, you came to the report and don't know what it's about.<\/p>\n<p><\/p>\n<p>Let me explain what will be discussed today. It so happens that our team has been involved in backups for quite some time. This is yet another talk in a series where we share how we store data safely, reliably, conveniently, and efficiently.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/4307e7b2acc60cd908d91e3f1b83a9b6.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In previous sessions, there have been many talks by Andrey Borodin and Vladimir Leskov. There were many of us. And we have been discussing WAL-G for many years. <\/p>\n<p><\/p>\n<p>clck.ru\/F8ioz \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/www.highload.ru\/moscow\/2018\/abstracts\/3964\">https:\/\/www.highload.ru\/moscow\/2018\/abstracts\/3964<\/a><\/noindex><\/p>\n<p><\/p>\n<p>clck.ru\/Ln8Qw \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/www.highload.ru\/moscow\/2019\/abstracts\/5981\">https:\/\/www.highload.ru\/moscow\/2019\/abstracts\/5981<\/a><\/noindex><\/p>\n<p><\/p>\n<p>This talk will differ a bit from the others in that those focused more on the technical aspect, while here I will discuss how we faced challenges associated with community growth. And how we came up with a small idea that helps us manage that. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/ec728f7d99ea9b6a5df4785d7e8a2a75.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A few years ago, WAL-G was a fairly small project that we inherited from Citus Data. We just took it over. It was developed by one person.<\/p>\n<p><\/p>\n<p>And there was only one thing that WAL-G lacked:<\/p>\n<p><\/p>\n<ul>\n<li>Backup from a replica.<\/li>\n<li>There were no incremental backups.<\/li>\n<li>There were no WAL-Delta backups. <\/li>\n<li>And a lot of other things were missing.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Over these few years, WAL-G has grown significantly.<\/strong> <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/16700dfcc2c54539c5a064894cd1ec88.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>By 2020, all of the above had been introduced. Additionally, we now have:<\/p>\n<p><\/p>\n<ul>\n<li>Over 1,000 stars on GitHub.<\/li>\n<li>150 forks.<\/li>\n<li>About 15 open PRs.<\/li>\n<li>And many contributors. <\/li>\n<li>And open issues constantly. This is despite the fact that we literally check in every day and do something about it. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/5e5dea14b328a7f0437017ca834d1eaf.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We concluded that this project requires more of our attention, even when we ourselves don't need to implement anything for our Managed Databases service at Yandex.<\/p>\n<p><\/p>\n<p>And somewhere in the autumn of 2018, an idea came to us. Normally, the team has a few ways to develop features or fix bugs when you lack manpower. For instance, you could hire another developer and pay them. Or you might bring in an intern for a while and pay them a salary too. But there\u2019s also a significant group of people, some of whom already know how to write code. It's just that you don\u2019t always know the quality of that code.<\/p>\n<p><\/p>\n<p>We thought about it and decided to try attracting students. However, students won\u2019t be involved in everything. They will do only a part of the work. For example, they might write tests, fix bugs, or implement features that don\u2019t affect the core functionality. Core functionality includes backup creation and recovery. If a bug occurs in backup creation, we risk data loss. And nobody wants that, of course. Everyone wants reliability. Therefore, we don't want to allow code we trust less than our own. In other words, any non-critical code is what we would like to receive from our additional hands.<\/p>\n<p><\/p>\n<p><strong>Under what conditions is a student\u2019s PR accepted?<\/strong><\/p>\n<p><\/p>\n<ul>\n<li>They are required to cover their code with tests. Everything must pass in CI. <\/li>\n<li>Additionally, we go through two reviews. One by Andrey Borodin and one by me. <\/li>\n<li>Moreover, to ensure it won't break anything in our service, I separately deploy the build with that commit. We verify with end-to-end tests that nothing is failing.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Special course on Open Source<\/strong><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/58832406f3acdf294cd68e6adde14ec8.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A little about why this is needed and why I think it's a cool idea. <\/p>\n<p><\/p>\n<p><strong>The benefit for us is obvious:<\/strong> <\/p>\n<p><\/p>\n<ul>\n<li>We get extra hands. <\/li>\n<li>And we look for team candidates among capable students who write solid code. <\/li>\n<\/ul>\n<p><\/p>\n<p><strong>What\u2019s the benefit for students?<\/strong><\/p>\n<p><\/p>\n<p>They may be less obvious, as students, at least, don\u2019t get paid for the code they write; they only receive grades. <\/p>\n<p><\/p>\n<p>I asked them about it. And from their words:<\/p>\n<p><\/p>\n<ul>\n<li>Experience as an Open Source contributor.<\/li>\n<li>To get a line in their CV.<\/li>\n<li>To showcase themselves and go through an interview at Yandex.<\/li>\n<li>To become a participant in GSoC.<\/li>\n<li>+1 special course for those who want to write code.<\/li>\n<\/ul>\n<p><\/p>\n<p>I won't go into how the course was structured. I will only say that WAL-G was the main project. Additionally, we included projects like Odyssey, PostgreSQL, and ClickHouse in this course.<\/p>\n<p><\/p>\n<p>And we provided tasks not only in this course but also issued diplomas and coursework.<\/p>\n<p><\/p>\n<p><strong>And what about the benefit for users?<\/strong><\/p>\n<p><\/p>\n<p>Now let's move on to the part that concerns you more. What is the benefit to you? The benefit is that students fixed many bugs and made feature requests that you asked us to implement. <\/p>\n<p><\/p>\n<p>And let me tell you about things that you have been wanting for a long time and which have been realized.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/bb836742bf4f99e1b91c891ed1802bf0.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Support for tablespaces. Support for tablespaces in WAL-G has probably been expected since the release of WAL-G, because WAL-G is a successor to another backup tool, WAL-E, which supported database backups with tablespaces.<\/p>\n<p><\/p>\n<p>Let me briefly remind you what this is and why it is needed. Typically, all your Postgres data resides in a single directory on the file system called the base directory. And this directory already contains all the files and subdirectories required by Postgres.<\/p>\n<p><\/p>\n<p>Tablespaces are directories that contain Postgres data, but they do not reside outside the base directory. The slide shows that tablespaces are outside the base directory.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/0f25591cfcaaeaec02903c9b98d8ac65.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>How does this look for Postgres? In the base directory, there is a separate subdirectory called pg_tblspc. Inside it are symlinks to the directories where Postgres data actually resides outside the base directory.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/9e9875696c03e6e42bb98b5a7a9acd75.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>When you use all this, these commands may look something like this for you. That is, you create a table in a specified tablespace and check where it currently resides. The last two lines, the two recently executed commands, indicate that there is a certain path. But in reality, this is not the actual path. This is a path with a prefix from the base directory to the tablespace. From there, it is matched with a symlink that points to your actual data.<\/p>\n<p><\/p>\n<p>We don't use all of this in our team, but many other WAL-E users used it and wrote to us that they wanted to switch to WAL-G, but this hindered them. Now it is supported.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/95ec319ccefa7e6c8bce193317407229.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Another feature that our special course brought us is catchup. People who have worked more with Oracle than with Postgres probably know about catchup.<\/p>\n<p><\/p>\n<p>A brief overview of what this is. This is how a cluster topology might typically look in our service. We have a master and a replica that streams the write-ahead log from it. The replica indicates to the master which LSN it is currently at. Simultaneously, the log might be archived. In addition to archiving the log, backups are also sent to the cloud, along with delta backups.<\/p>\n<p><\/p>\n<p>What could be the problem? When you have a fairly large database, it can happen that your replica starts to lag significantly behind the master. It lags so much that it can never catch up. This problem usually needs to be addressed somehow. <\/p>\n<p><\/p>\n<p>The simplest way to resolve this is to remove the replica and reinitialize it since it will never catch up, and we need to deal with the problem. However, this process is quite lengthy because restoring a full backup of a 10 TB database is very time-consuming. We would like to do this as quickly as possible when such issues arise. This is precisely what catchup is designed for.<\/p>\n<p><\/p>\n<p>Catchup allows you to use delta backups that are stored in the cloud in this way. You specify the LSN of the lagging replica and provide it in the catchup command to create a delta backup between that LSN and the LSN where your cluster currently is. After that, you restore this backup to the replica that is lagging.<\/p>\n<p><\/p>\n<p><strong>Other databases<\/strong><\/p>\n<p><\/p>\n<p>Additionally, our students brought us a number of features. Since we at Yandex work with not only Postgres, but also MySQL, MongoDB, Redis, and ClickHouse, at one point we needed to ensure we could perform backups with point-in-time recovery for MySQL, and have the capability to upload them to the cloud.<\/p>\n<p><\/p>\n<p>We wanted to do this in a way similar to how WAL-G operates. We decided to experiment and see how it would all turn out.<\/p>\n<p><\/p>\n<p>Initially, without separating this logic, we wrote the code in a fork. We saw that we had some working model, and it was feasible. Later, we thought that our main community consists of Postgres users who utilize WAL-G. Therefore, we needed to separate these components. That is to say, when we modify the code for Postgres, we do not break MySQL, and vice versa.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/deef7615eaeb0b8bd790fb253cc42312.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The initial idea for separating this was to use the same approach as that used in PostgreSQL extensions. Essentially, to back up MySQL, you had to install some dynamic library. <\/p>\n<p><\/p>\n<p>However, the asymmetry of this approach is immediately evident. When backing up Postgres, you simply install a standard backup tool for Postgres and everything works fine. But for MySQL, it turns out that you install a backup tool for Postgres and also add a dynamic library for MySQL. That sounds a bit strange. We thought so too and decided that this is not the solution we need. <\/p>\n<p><\/p>\n<p><strong>Different builds for Postgres, MySQL, MongoDB, Redis<\/strong><\/p>\n<p><\/p>\n<p>But this allowed us, as we believe, to arrive at the right solution \u2013 to allocate different builds for different databases. This enabled us to isolate the logic tied to backups of various databases, which would interface with a common API implemented by WAL-G.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/6c810d5f065de06fc200bdd028707318.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>This is the part we wrote ourselves \u2013 before giving students the tasks. That is, this is precisely where they could potentially make mistakes, so we decided it was better to handle it ourselves to ensure everything would be done correctly. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/3a864a85bfb820ef64cbacc029a9198c.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>After that, we issued the tasks. They were immediately taken up. Students were required to support three databases. <\/p>\n<p><\/p>\n<p>This is MySQL, which we have been backing up with WAL-G in this way for over a year now. <\/p>\n<p><\/p>\n<p>And now MongoDB is nearing production, where it is being fine-tuned. Essentially, we wrote the framework for all this. Then the students created some working components, and subsequently, we bring them to a state that we can accept in our production.<\/p>\n<p><\/p>\n<p>These tasks did not imply that students had to write complete backup tools for each of these databases. We were not faced with such a problem. Our issue was that we wanted point-in-time recovery and we wanted to back up to the cloud. We asked the students to write some code that would solve this. The students utilized already existing backup tools that somehow create backups, and then they integrated all this with WAL-G, which redirected everything to the cloud. They also added point-in-time recovery to this.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/7494aef259c7599e8bf0b70d77e1bb09.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>What else did the students contribute? They brought support for Libsodium encryption to WAL-G.<\/p>\n<p><\/p>\n<p>We also established backup retention policies. Now backups can be marked as permanent. This makes it more convenient for your service to automate the process of their storage. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/6e932ab13dbc3119f7f1da1122443b10.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>What were the results of this experiment?<\/p>\n<p><\/p>\n<p>Initially, over 100 people registered for the course. I didn't mention at first that the university in Yekaterinburg is the Ural Federal University. That's where we made all the announcements. 100 people registered. However, much fewer actually started working, around 30 people. <\/p>\n<p><\/p>\n<p>Even fewer people completed the course, as it required writing tests for existing codes. They also needed to fix some bugs or implement certain features. A portion of the students still managed to complete the course.<\/p>\n<p><\/p>\n<p>Currently, students have fixed about 14 issues for this course and created 10 features of various sizes. It seems to me that this effectively replaces one or two developers. <\/p>\n<p><\/p>\n<p>Additionally, we issued diplomas and course projects. 12 students received diplomas, 6 of them have already defended their projects with top marks. The others haven't defended yet, but I believe they will do well too. <\/p>\n<p><\/p>\n<p>Plans for the future<\/p>\n<p><\/p>\n<p>What are our future plans?<\/p>\n<p><\/p>\n<p>At the very least, we want to implement the feature requests we've already heard from users. These include:<\/p>\n<p><\/p>\n<ul>\n<li>Monitoring the accuracy of the timeline tracking in the HA-cluster backup archive. This can be done using WAL-G. I believe we can find students willing to take on this task. <\/li>\n<li>We already have a person responsible for transferring backups and WAL between clouds. <\/li>\n<li>We recently published an idea that we can further accelerate WAL-G by unpacking incremental backups without overwriting pages and optimizing the archives we send there. <\/li>\n<\/ul>\n<p><\/p>\n<p>Feel free to share your thoughts here<\/p>\n<p><\/p>\n<p>What was the purpose of this report? It\u2019s to highlight that now, besides the four of us maintaining this project, we have additional hands that are quite numerous, especially if you message them directly. If you are backing up your data using WAL-G, or would like to switch to it, we can easily accommodate your wishes.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/7b0dc1633121ea6493610b08f68a14d2.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Here is the QR code and link. You can use them to share all your requests. For example, if there\u2019s a bug we haven\u2019t fixed, or a feature you really want that isn\u2019t available in any of the backup solutions, including ours. Please make sure to mention it.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"WAL-G: new capabilities and community expansion. Georgiy Rylov\" src=\"\/wp-content\/uploads\/2020\/04\/56f1c4eef8522d2d6628ea98b1278846.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Questions<\/p>\n<p><\/p>\n<p><em>Hello! Thank you for the presentation! I have a question about WAL-G, but not about Postgres. WAL-G backs up MySQL and triggers an extra backup. If we take modern installations on CentOS and if you do yum install MySQL, it will install MariaDB. Since version 10.3, extra backup is not supported; only MariaDB backup is supported. How are you handling this?<\/em><\/p>\n<p><\/p>\n<p>Currently, we haven't tried backing up MariaDB. We had requests for FoundationDB support, but in general, if there is such a request, we can find people to do it. It\u2019s not as time-consuming and complicated as I thought it would be.<\/p>\n<p><\/p>\n<p><em>Good day! Thank you for the presentation! I have a question about potentially new features. Are you ready to make WAL-G work with tapes to enable tape backups?<\/em><\/p>\n<p><\/p>\n<p>Are you referring to backup on tape storage?<\/p>\n<p><\/p>\n<p><em>Yes.<\/em><\/p>\n<p><\/p>\n<p>There is Andrey Borodin there, who can answer this question better than I can. <\/p>\n<p><\/p>\n<p>(Andrey) Yes, thanks for the question! We had a request to transfer backups to tape from cloud storage. And for this, <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wal-g\/wal-g\/pull\/596\">we're working on<\/a><\/noindex> the transfer between clouds. Because transferring between clouds is a generalized version of tape transfer. Additionally, we have a scalable architecture in terms of storage. By the way, many storages were written by students. And if you write a storage for tape, it will definitely be supported. We are ready to consider pull requests. You need to write a file and read a file. If you implement these in Go, it usually takes about 50 lines of code. Then tape will be supported in WAL-G. <\/p>\n<p><\/p>\n<p><em>Thank you for the presentation! An interesting development process. Backup is a significant part of functionality that should be well covered by tests. When you implemented functionality for new databases, were tests written by students as well, or did you write the tests yourself and then hand the implementation over to the students?<\/em><\/p>\n<p><\/p>\n<p>Tests were also written by students. But the students wrote more for features like new databases. They wrote integration tests and unit tests. If integration tests pass, i.e., at the moment \u2014 this is a scenario that you execute manually or that cron does for you, for example. So the scenario is quite clear. <\/p>\n<p><\/p>\n<p><em>Students do not have much experience. How much time does it take for reviews?<\/em><\/p>\n<p><\/p>\n<p>Yes, reviewing takes a significant amount of time. That is, usually when several committers come in and say, 'I did this, I did that', one needs to think and allocate at least half a day to figure out what they have written. Because the code needs to be read carefully. They didn't go through an interview. We don't know them very well, so this takes substantial time.<\/p>\n<p><\/p>\n<p><em>Thank you for the presentation! Earlier, Andrey Borodin stated that the archive_command in WAL-G should be called directly. But in the case of some patron cluster, we need additional logic to determine the node from which to send the wals. How do you solve this problem on your end?<\/em><\/p>\n<p><\/p>\n<p>What issue are you facing here? For example, do you have a synchronous replica from which you are taking a backup? Or what?<\/p>\n<p><\/p>\n<p>(Andrey) The fact is that WAL-G is indeed intended to be used without wrapping it in shell scripts. If something is missing, then let's write the logic that should be inside WAL-G. Regarding where the archiving should come from, we believe that archiving should be from the current master in the cluster. Archiving from a replica is a bad idea. There can be various scenarios with problems. In particular, issues with archiving timelines and other additional information. Thanks for the question!<\/p>\n<p><\/p>\n<p>(Clarification: We dropped the wrapping in shell scripts. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wal-g\/wal-g\/issues\/549\">in this issue<\/a><\/noindex>)<\/p>\n<p><\/p>\n<p><em>Good evening! Thank you for the presentation! I was intrigued by the catchup feature you described. Have you ever encountered a situation where a replica fell behind and could not catch up? I did not find a description of this feature in the WAL-G documentation.<\/em><\/p>\n<p><\/p>\n<p>Catchup appeared literally in the 20s of January 2020. Perhaps it would be worth working on the documentation a bit more. We ourselves are writing it and not necessarily doing it perfectly. And we might need to start requiring students to write it as well.<\/p>\n<p><\/p>\n<p><em>Is it already released?<\/em><\/p>\n<p><\/p>\n<p>The pull request has been merged, that is, I checked it. I tried it on a test cluster. So far, we haven't had a situation where we could test this in a production example.<\/p>\n<p><\/p>\n<p><em>When to expect it?<\/em><\/p>\n<p><\/p>\n<p>I don\u2019t know. Wait a month, we\u2019ll definitely check.<\/p>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/498428\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430\u0447\u0430\u043b\u0430 2020 \u0433\u043e\u0434\u0430 \u0413\u0435\u043e\u0440\u0433\u0438\u044f \u0420\u044b\u043b\u043e\u0432\u0430 &quot;WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430&quot; \u0423 \u043c\u0435\u0438\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 open-source \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u043f\u043e \u043c\u0435\u0440\u0435 \u0438\u0445 \u0440\u043e\u0441\u0442\u0430. \u041a\u0430\u043a \u043f\u0438\u0441\u0430\u0442\u044c \u0432\u0441\u0435 \u0431\u043e\u043b\u044c\u0448\u0435 \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u044b\u0445 \u0444\u0438\u0447, \u0447\u0438\u043d\u0438\u0442\u044c \u0432\u0441\u0435 \u0431\u043e\u043b\u044c\u0448\u0435 issues&#8217;\u043e\u0432 \u0438 \u0443\u0441\u043f\u0435\u0432\u0430\u0442\u044c \u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c \u0432\u0441\u0435 \u0431\u043e\u043b\u044c\u0448\u0435 pull request&#8217;\u043e\u0432? \u041d\u0430 \u043f\u0440\u0438\u043c\u0435\u0440\u0435 WAL-G(backup-tool for PostgreSQL) \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443 \u043f\u0440\u043e \u0442\u043e, \u043a\u0430\u043a \u043c\u044b \u0440\u0435\u0448\u0430\u043b\u0438 \u044d\u0442\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":78888,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-78887","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430\u0447\u0430\u043b\u0430 2020 \u0433\u043e\u0434\u0430 \u0413\u0435\u043e\u0440\u0433\u0438\u044f \u0420\u044b\u043b\u043e\u0432\u0430 &quot;WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430&quot; \u0423 \u043c\u0435\u0438\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 open-source \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u043f\u043e.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430. \u0413\u0435\u043e\u0440\u0433\u0438\u0439 \u0420\u044b\u043b\u043e\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430\u0447\u0430\u043b\u0430 2020 \u0433\u043e\u0434\u0430 \u0413\u0435\u043e\u0440\u0433\u0438\u044f \u0420\u044b\u043b\u043e\u0432\u0430 &quot;WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430&quot; \u0423 \u043c\u0435\u0438\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 open-source \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u043f\u043e.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-04-22T17:42:51+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-04-22T17:42:51+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47WAL-G: new features and community expansion. Georgy Rylov | ProHoster","description":"I suggest reviewing the transcript of Georgy Rylov's talk from early 2020, \"WAL-G: New Features and Community Expansion.\" Open-source maintainers face numerous issues.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430. \u0413\u0435\u043e\u0440\u0433\u0438\u0439 \u0420\u044b\u043b\u043e\u0432 | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430\u0447\u0430\u043b\u0430 2020 \u0433\u043e\u0434\u0430 \u0413\u0435\u043e\u0440\u0433\u0438\u044f \u0420\u044b\u043b\u043e\u0432\u0430 &quot;WAL-G: \u043d\u043e\u0432\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u0435 \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0430&quot; \u0423 \u043c\u0435\u0438\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 open-source \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u043f\u043e.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/wal-g-novye-vozmozhnosti-i-rasshirenie-soobshhestva-georgij-rylov","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-04-22T17:42:51+00:00","article:modified_time":"2020-04-22T17:42:51+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"78887","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:49:45","updated":"2022-09-29 18:48:15","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/78887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=78887"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/78887\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/78888"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=78887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=78887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=78887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}