Replication is not a backup. Or is it? Here's how we used delayed replication for recovery after accidentally deleting labels.
at GitLab are responsible for the operation of the largest GitLab instance in existence. It hosts 3 million users and nearly 7 million projects, making it one of the largest open-source SaaS sites with dedicated architecture. Without a PostgreSQL database system, GitLab.com wouldn't get very far, and we do a lot to ensure resilience in case of failures that could lead to data loss. While such a disaster is unlikely, we are well-prepared and equipped with various backup and replication mechanisms.
Replication is not a means of backing up databases (). But now we'll see how quickly we can recover accidentally deleted data using delayed replication: I user for the project and lost connection with merge requests and issues.
With the delayed replica, we recovered the data in just 1.5 hours. Here's how it went.
Point-in-Time Recovery with PostgreSQL
PostgreSQL has a built-in feature that restores the database state to a specific point in time. It's called (PITR) and uses the same mechanisms that keep the replica up-to-date: starting from a reliable snapshot of the entire database cluster (base backup), we apply a series of state changes up to a specified point in time.
To use this feature for cold backups, we regularly create a base backup of the database and store it in an archive (GitLab archives live in ). We also track database state changes by archiving the write-ahead log (, WAL). With all this, we can perform PITR for disaster recovery: starting from a snapshot made before the error, we apply changes from the WAL archive up to the failure.
What is delayed replication?
Delayed replication is the application of changes from the WAL with a delay. That is, the transaction occurred at the hour X, but it will appear in the replica with a delay of d the hour X + d.
In PostgreSQL, there are 2 ways to set up a physical database replica: recovery from archive and streaming replication. , in essence, works like PITR, but continuously: we constantly extract changes from the WAL archive and apply them to the replica. A directly pulls the WAL stream from the higher-level database host. We prefer restoration from the archive—it is easier to manage and has normal performance that keeps pace with the working cluster.
How to set up delayed recovery from the archive
are described in the file recovery.conf. Example:
standby_mode = 'on'
restore_command = '\/usr\/bin\/envdir \/etc\/wal-e.d\/env \/opt\/wal-e\/bin\/wal-e wal-fetch -p 4 "%f" "%p"'
recovery_min_apply_delay = '8h'
recovery_target_timeline = 'latest'With these parameters, we set up a delayed replica with recovery from the archive. It uses to extract WAL segments (restore_command) from the archive, and changes will be applied after eight hours (recovery_min_apply_delay). The replica will track the changes on the timeline in the archive, for example, due to failover in the cluster (recovery_target_timeline).
C recovery_min_apply_delay You can set up streaming replication with delay, but there are a couple of pitfalls related to replication slots, hot standby feedback, and so on. The WAL archive helps avoid them.
Parameter recovery_min_apply_delay was introduced only in PostgreSQL 9.3. In earlier versions, to set up delayed replication, you needed to configure a combination of (pg_xlog_replay_pause(), pg_xlog_replay_resume()) or keep WAL segments in the archive during the delay period.
How does PostgreSQL do this?
It is interesting to see how PostgreSQL implements delayed recovery. Let's take a look at . It is called from for each record from the WAL.
static bool
recoveryApplyDelay(XLogReaderState *record)
{
uint8 xact_info;
TimestampTz xtime;
long secs;
int microsecs;
/* nothing to do if no delay configured */
if (recovery_min_apply_delay <= 0)
return false;
/* no delay is applied on a database not yet consistent */
if (!reachedConsistency)
return false;
/*
* Is it a COMMIT record?
*
* We deliberately choose not to delay aborts since they have no effect on
* MVCC. We already allow replay of records that don't have a timestamp,
* so there is already opportunity for issues caused by early conflicts on
* standbys.
*/
if (XLogRecGetRmid(record) != RM_XACT_ID)
return false;
xact_info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
if (xact_info != XLOG_XACT_COMMIT &&
xact_info != XLOG_XACT_COMMIT_PREPARED)
return false;
if (!getRecordTimestamp(record, &xtime))
return false;
recoveryDelayUntilTime =
TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay);
/*
* Exit without arming the latch if it's already past time to apply this
* record
*/
TimestampDifference(GetCurrentTimestamp(), recoveryDelayUntilTime,
&secs, µsecs);
if (secs <= 0 && microsecs <= 0)
return false;
while (true)
{
// Shortened:
// Use WaitLatch until we reached recoveryDelayUntilTime
// and then
break;
}
return true;
}The essence is that the delay is based on the physical time recorded in the commit timestamp of the transaction (xtime). As you can see, the delay is applied only to commits and does not affect other records — all changes are applied directly, and the commit is postponed, so we will see the changes only after the configured delay.
How to use delayed replica for data recovery
Suppose we have a production database cluster and a replica with an eight-hour delay. Let’s see how to recover data using the example of .
When we found out about the issue, we for the delayed replica:
SELECT pg_xlog_replay_pause();With the pause, we had no risk that the replica would repeat the request DELETE. A useful thing if you need time to sort everything out.
The point is that the delayed replica must reach the moment before the request DELETE. We roughly knew the physical time of the deletion. We deleted recovery_min_apply_delay and added recovery_target_time downward API support (simultaneously with this in recovery.conf. This way, the replica reaches the required moment without delays:
recovery_target_time = '2018-10-12 09:25:00+00'With timestamps, it’s better to subtract a little excess to avoid mistakes. However, the more you subtract, the more data you lose. Again, if we miss the request DELETE, everything will be deleted again and we'll have to start over (or take a cold backup for PITR).
We restarted the delayed Postgres instance, and the WAL segments were replayed up to the specified time. You can track progress at this stage with the following query:
SELECT
-- current location in WAL
pg_last_xlog_replay_location(),
-- current transaction timestamp (state of the replica)
pg_last_xact_replay_timestamp(),
-- current physical time
now(),
-- the amount of time still to be applied until recovery_target_time has been reached
'2018-10-12 09:25:00+00'::timestamptz - pg_last_xact_replay_timestamp() as delay;If the timestamp no longer changes, recovery is complete. You can configure the action , to either stop, promote, or pause the instance after replay (by default, it pauses).
The database has reverted to the state before that unfortunate query. Now, you can, for example, export the data. We exported the removed shortcut data and all relationships with tasks and merge requests and moved them to the working database. If the loss is significant, you can simply promote the replica and use it as the primary. However, this will cause all changes made after the point to which we restored to be lost.
Instead of timestamps, it's better to use transaction IDs. It's useful to record these IDs, for example, for DDL operators (like DROP TABLE), using log_statements = 'ddl'. If we had the transaction ID, we would take recovery_target_xid and replay everything up to the transaction before the query. DELETE.
Returning to work is very simple: remove all changes from recovery.conf and restart Postgres. Soon there will again be an eight-hour delay in the replica, and we will be ready for future troubles.
Advantages of recovery
With a delayed replica instead of a cold backup, there's no need to spend hours restoring the entire snapshot from the archive. For example, it takes us five hours to retrieve the entire base backup of 2 TB. Then we would also have to apply all the daily WAL to recover to the desired state (in the worst case).
The delayed replica is superior to a cold backup for two reasons:
- There’s no need to retrieve the entire base backup from the archive.
- There’s a fixed eight-hour window of WAL segments to replay.
We also constantly check if we can perform PITR from the WAL, and we would quickly notice any damages or other issues with the WAL archive by monitoring the lag of the delayed replica.
In this example, it took us 50 minutes to recover, meaning the speed was 110 GB of WAL data per hour (the archive was still on ). We resolved the issue and restored the data in 1.5 hours.
Summary: Where delayed replication is useful (and where it isn't)
Use delayed replication as a first aid tool if you accidentally lose data and notice the problem within the configured delay.
But keep in mind: replication is not a backup.
Backups and replication have different purposes. A cold backup will be useful if you accidentally made DELETE or DROP TABLE. We create a backup from cold storage and restore the previous state of the table or the entire database. However, during this process, the DROP TABLE is almost instantly replicated across all replicas in the working cluster, so standard replication won't help here. Replication itself keeps the database available when individual servers fail and distributes the load.
Even with a delayed replica, we sometimes really need a cold backup in a safe place in case of a data center failure, hidden corruption, or other events that you might not notice immediately. Here, replication alone isn't sufficient.
Note. On we are currently protecting against data loss only at the system level and are not restoring data at the user level.
Source: habr.com
