Each version of Firebird has its own version of the database disk structure format – O(n)D(isk)S(tructure). Up to and including version 2.5, the Firebird engine could work with ODS from previous versions, meaning databases from older versions would open in the new version and operate in compatibility mode. However, the Firebird 3.0 engine only works with databases in its own ODS version 12.0.
To upgrade to 3.0, the database from 2.5 must be converted to the new format using backup/restore. Of course, we assume that the database has been prepped for conversion—i.e., the metadata and queries have been checked for compatibility with Firebird 3.0.
If you follow the standard approach, it means you need to back up on version 2.5, then install 3.0 and perform the restore. This procedure is acceptable if there is enough time, but when migrating large databases or simultaneously migrating several dozen databases, where time is tight, you can use streaming conversion, which is 30-40% faster. Read below for how to do this (on Windows and Linux).
The general idea is to use a pipeline for acceleration:
gbak -b … database25 stdout | gbak -c … stdin database30Gbak from 2.5 generates a backup in a linear format and directs it to stdout, which is immediately picked up via stdin by gbak from 3.0 to create a new database.
You must organize such a pipeline using local (file) access methods, as network access (even through localhost) will significantly slow down the process.
Below we discuss the details for Windows and Linux.
Windows
In the case of Windows, the easiest way is to create a completely standalone build of Firebird. For this you take , rename fbemded.dll to fbclient.dll, add utilities gbak.exe and (optionally) isql.exe from the regular 2.5 archive.
Firebird 3.0 uses and requires no further modifications.
The minimal option (that does not require the installation of VS2008/VS2010 runtime libraries on the target system) contains the following files:
25/gbak.exe
25/fbclient.dll
25/firebird.conf
25/firebird.log
25/firebird.msg
25/ib_util.dll
25/icudt30.dll
25/icuin30.dll
25/icuuc30.dll
25/Microsoft.VC80.CRT.manifest
25/msvcp80.dll
25/msvcr80.dll
30/fbclient.dll
30/firebird.conf
30/firebird.msg
30/gbak.exe
30/ib_util.dll
30/icudt52.dll
30/icudt52l.dat
30/icuin52.dll
30/icuuc52.dll
30/msvcp100.dll
30/msvcr100.dll
30/intl/fbintl.conf
30/intl/fbintl.dll
30/plugins/engine12.dllAn experienced administrator may notice that the files intl/fbintl.dll and intl/fbintl.conf are not included in 2.5. This is indeed the case, as gbak does not use connection character sets and does not convert data between character sets, but on the 'receiving' side of Firebird 3.0, these files are necessary when creating indexes.
In firebird.conf, it is recommended to add the following for Firebird 3.0:
MaxUnflushedWrites = -1
MaxUnflushedWriteTime = -1Also, it is advisable to set different values for IpcName for 2.5 and 3.0.
When choosing values for other parameters in firebird.conf, we proceed from a simple consideration: during the data transfer stage, one process runs gbak 2.5, and another runs 3.0; then 2.5 completes its work and 3.0 begins to build indexes.
To speed up the index building phase in 3.0, it is recommended to increase the TempCacheLimit parameter size to about 40% of RAM (if it is a dedicated server, of course).
For example, if the server has 16 GB of RAM, you could set
TempCacheLimit=6GOf course, such a value can only be set on 64-bit Firebird 3, since any 32-bit process cannot allocate more than 2 gigabytes of memory.
For 2.5, this parameter does not need to be changed – it cannot exceed 2 gigabytes, and it does not affect speed during backup.
Before performing the operation, it is necessary to check that the page cache in the database header is set to 0 (command gstat -h databasename, see the line Page buffers).
If the cache is explicitly set in the DB header, it overrides the values from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values, it can lead to excessive memory consumption and swapping.
Next, copy the files to the target system.
The conversion is performed after stopping the 'system' service of Firebird 2.5, in the command line with elevated rights to a local administrator (example):
set ISC_USER=owner
"25/gbak" -z -b -g -v -st t -y 25.log database25 stdout|^
"30/gbak" -z -c -v -st t -y 30.log stdin database30In this example, a 'forward slash' is used in quotes (a valid 'unix-style'), and the 'caret' (the '^' character) escapes the newline character, which is convenient for typing long commands. The -st(atus) option appeared in Firebird 2.5.8 and allows logging of the gbak process's runtime data (details – in the documentation).
Linux
On Linux, Firebird 3 depends on the tommath library. In CentOS (RHEL), this library is located in the epel repository; in Ubuntu (Debian), it is in the system.
For CentOS, you need to first enable the epel repository and then proceed to
yum install libtommathUbuntu does not require additional repositories, but Ubuntu 16 and Ubuntu 18 install different package versions – libtommath0 and libtommath1, respectively.
Firebird 3.0 looks for tommath.so.0, and for Ubuntu 18, you also need to create a symlink from tommath.so.0 to tommath.so.1. First, you need to locate tommath.so.1.
The required path in Ubuntu is /usr/lib/x86_64-linux-gnu/, but it may vary in other Debian-based distributions.
The second issue is that up to Firebird 3.0.1, there was no straightforward way to install two different server versions. The option "compile from source with the required prefix" is not considered due to its relative complexity.
For Firebird 3.0.2 and later, a and a separate installer option (-path path) has been implemented.
Assuming that the tommath library and, if necessary, the symlink for tommath.so.0 are added to the system, you can install the current (as of the writing of this article) distribution of Firebird 3.0.4 in, for example, /opt/fb3:
./install.sh -path /opt/fb3After that, you can stop the Firebird system service and start the stream conversion.
When stopping Firebird, keep in mind that Firebird 2.5 processes in Classic mode are usually started by xinetd – therefore, you either need to disable the firebird service for xinetd or stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they only work on Windows) and change Firebird 2.5 settings.
In Linux, local (file) access to Firebird 2.5 is not equivalent to the embedded option under Windows – server 2.5 will work in the gbak process (without the network part), but access rights will be checked by the user database, meaning that both a login and a password will be required:
export ISC_USER=username ISC_PASSWORD=password
/opt/firebird/bin/gbak -b … base25 stdout
|/opt/fb3/bin/gbak -c … stdin base30After successful conversion, you should first remove the "additional" Firebird 3.0, then the "main" Firebird 2.5, and only after that perform a clean installation of Firebird 2.5 — preferably from the standard tar.gz installer, not through repositories, as the version in repositories may be outdated.
Also, after restoring the DB on Linux and reinstalling, make sure that the new DB has the user firebird as its owner.
If not, you will need to correct it.
chown firebird.firebird databaseSummary
In addition to saving time and disk space, streaming conversion has another important advantage – the database transformation occurs without deleting the existing Firebird 2.5, which significantly simplifies the rollback in case of a failed conversion (most often due to lack of space or an unexpected reboot during the migration process).
Time savings are associated with the fact that 'classic' conversion includes 'backup time' plus 'recovery time'. Recovery consists of two parts: reading data from the backup file and building the index.
With streaming conversion, the total time is calculated as 'backup time plus five to ten percent' and 'index building time'.
Specific results depend on the structure of the database, but on average, recovery time is approximately double that of backup time. Therefore, if we take the backup time as one unit, the 'classic conversion' takes three units of time, while streaming takes two units of time. Additionally, increasing TempCacheLimit helps reduce time further.
Overall, streaming conversion in practice allows for a savings of 30-40% of the time compared to sequential backup and restore.
Questions?
Please direct all questions in the comments, or send them to the author of the methodology and co-author of this article – Vasiliy Sidorov, leading system engineer at 'iBase', at bs at ibase ru.
Only registered users can participate in the survey. , please.
Which version of Firebird are you using?
Firebird 3.x
Firebird 2.5
Firebird 2.1
Firebird 2.0, 1.5 or 1.0
16 users voted. 1 user abstained.
Source: habr.com
