Each version of Firebird has its own version of the database disk structure format, O(n)D(isk)S(tructure). Up to version 2.5 inclusive, the Firebird engine could work with ODS of previous versions, that is, databases from old versions were opened by the new version and worked in compatibility mode, but the Firebird 3.0 engine only works with databases in its own ODS version 12.0.
To migrate to 3.0, the database from 2.5 must be converted to the new format via backup/restore. Of course, we assume that the database was previously prepared for conversion - i.e. metadata and queries have been checked for compatibility with Firebird 3.0.
If you follow the standard approach, this means that you need to make a backup on version 2.5, then install 3.0 and make a restore. Such a procedure is acceptable if there is enough time, but when migrating large databases, or when migrating several dozen databases at the same time, when time is running out, you can use streaming conversion, which is 30-40% faster. How exactly to do this (under Windows and under Linux), read under the cut.
The general idea is that we will use a pipeline to speed things up:
gbak -b … база25 stdout | gbak -c … stdin база30Gbak from 2.5 generates a backup in a linear format and sends it to stdout, which immediately picks up gbak from 3.0 via stdin and creates a new database.
It is necessary to organize such a pipeline with a local (file) access method, since network access (even through localhost) will significantly slow down the process.
We go over the details for Windows and Linux below.
Windows
In the case of Windows, the easiest way is to make a completely standalone build of Firebird. For this we take , rename fbemded.dll to fbclient.dll, add gbak.exe and (optionally) isql.exe utilities from the “regular” 2.5 archive.
Firebird 3.0 uses and does not require any modification.
The most minimal version (which does not require installation of the VS2008/VS2010 runtime libraries on the target system) contains the following files:
25/gbak.exe
25/fbclient.dll
25/firebird.conf
25/firebird.log
25/firebird.msg
25/ib_util.dll
25/icudt30.dll
25/icuin30.dll
25/icuuc30.dll
25/Microsoft.VC80.CRT.manifest
25/msvcp80.dll
25/msvcr80.dll
30/fbclient.dll
30/firebird.conf
30/firebird.msg
30/gbak.exe
30/ib_util.dll
30/icudt52.dll
30/icudt52l.dat
30/icuin52.dll
30/icuuc52.dll
30/msvcp100.dll
30/msvcr100.dll
30/intl/fbintl.conf
30/intl/fbintl.dll
30/plugins/engine12.dllAn experienced administrator may notice that 2.5 does not include the intl/fbintl.dll and intl/fbintl.conf files. This is true, since gbak does not use a connection charset and does not convert data between charsets, but on the "receiving" side of Firebird 3.0, these files are necessary when creating indexes.
In firebird.conf Firebird 3.0 is recommended to add:
MaxUnflushedWrites = -1
MaxUnflushedWriteTime = -1Also, it is desirable to set different IpcName values for 2.5 and 3.0.
When choosing the values of other parameters of firebird.conf, we proceed from a simple consideration: at the data transfusion stage, gbak runs 2.5 in one process, and 3.0 in the other, then 2.5 exits, and 3.0 starts building indexes.
To speed up the index building phase in 3.0, it is recommended to increase the size of the TempCacheLimit parameter to ~40% RAM (if it is a dedicated server, of course).
For example, if the server has 16 GB of RAM, then you can put
TempCacheLimit=6GOf course, this value can be set only for 64-bit Firebird 3, since any 32-bit process cannot allocate more than 2 gigabytes of memory.
In 2.5, this parameter does not need to be changed - it cannot be more than 2 gigabytes anyway, and it does not affect the speed during backup.
Before performing the operation, you need to check that the page cache in the database header is set to 0 (command gstat -h databasename, see the Page buffers line).
If the cache is set explicitly in the database header, then it overrides the values from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values, it can lead to excessive memory consumption and swapping.
Next, copy the files to the target system.
The conversion is carried out after stopping the "system" Firebird 2.5 service, on the command line with elevated rights to the local administrator (example):
set ISC_USER=владелец
"25/gbak" -z -b -g -v -st t -y 25.log база25 stdout|^
"30/gbak" -z -c -v -st t -y 30.log stdin база30This example uses a "forward slash" in quotes (valid "unix-style"), and a "hat" (the "^" character) escapes the newline character, which is useful when typing long commands. The -st(atus) option appeared in Firebird 2.5.8 and allows you to log data about the time the gbak process was running (for details, see the documentation).
Linux
On Linux Firebird 3 depends on the tommath library. On CentOS (RHEL) this library is located in the epel repository, on Ubuntu (Debian) in the system repository.
For CentOS, you must first connect the epel repository and only then do
yum install libtommathUbuntu does not need to include additional repositories, but Ubuntu 16 and Ubuntu 18 install different versions of the packages - libtommath0 and libtommath1, respectively.
Firebird 3.0 looks for tommath.so.0 and for Ubuntu 18 it is additionally required to create a link (symlink) from tommath.so.0 to tommath.so.1. To do this, you first need to find tommath.so.1.
Searched path in Ubuntu - /usr/lib/x86_64-linux-gnu/, but other Debian-based distributions may be different.
The second problem is related to the fact that up to and including Firebird 3.0.1, there was no easy way to install two different server versions. We do not consider the option “compile from source with the required prefix” because of its relative complexity.
For Firebird 3.0.2 and higher implemented and a separate installer option (-path path).
Assuming that the tommath library and, if necessary, a symlink for tommath.so.0 have been added to the system, you can install the current (at the time of this writing) Firebird 3.0.4 distribution in, for example, /opt/fb3:
./install.sh -path /opt/fb3After that, you can stop the Firebird system service and start streaming conversion.
When stopping Firebird, keep in mind that Firebid 2.5 processes in Classic mode are usually started by xinetd - so you need to either disable the firebird service for xinetd or stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they only work on Windows) and change Firebird 2.5 settings.
In Linux, local (file) access of Firebird 2.5 is not equivalent to the embeded version under Windows - the 2.5 server will work in the gbak process (without the network part), but access rights will be checked against the user base, which means that not only a login, but also a password will be required :
export ISC_USER=username ISC_PASSWORD=password
/opt/firebird/bin/gbak -b … база25 stdout
|/opt/fb3/bin/gbak -c … stdin база30After a successful conversion, you must first uninstall the "additional" Firebird 3.0, then the "main" Firebird 2.5, and after that perform a clean installation of Firebird 2.5 - and it's best from the regular tar.gz installer, and not through the repositories, because. the version in the repositories may lag behind.
Also, after restoring the database on Linux and reinstalling, you need to check that the new database is owned by the firebird user.
If this is not the case, then it will need to be corrected.
chown firebird.firebird databaseСonclusion
In addition to saving time and disk space, streaming conversion has another important advantage - database conversion is done without deleting the existing Firebird 2.5, which greatly simplifies rollback in case of unsuccessful conversion (most often due to lack of space or unexpected reboot during the migration process).
The time saving is due to the fact that the "classic" conversion is "backup time" plus "restore time". Recovery consists of two parts: reading data from a backup file and building an index.
With streaming conversion, the total time is obtained as “backup time plus five to ten percent” and “index building time”.
Specific results depend on the structure of the database, but on average, the recovery time is approximately twice the backup time. Therefore, if we take backup time as a unit, then “classic conversion” is three units of time, streaming is two units of time. Increasing TempCacheLimit helps to further reduce the time.
In general, streaming conversion in practice allows you to save 30-40% of the time of alternate backup and restoring.
Questions?
Please write all questions in the comments, or send them to the author of the methodology and co-author of this article - Vasily Sidorov, iBase Leading System Engineer, at bs at ibase ru.
Only registered users can participate in the survey. , you are welcome.
What version of Firebird are you using?
-
Firebird 3.x
-
Firebird 2.5
-
Firebird 2.1
-
Firebird 2.0, 1.5 or 1.0
16 users voted. 1 user abstained.
Source: habr.com