The linux-api mailing list has presented a proposal (RFC) to replace the outdated binary formats of system logs lastlog, btmp, utmp, and wtmp with new shared libraries that use SQLite as a backend. This initiative aims to address accumulated issues, including the overflow of 32-bit time counters in 2038, lack of extensibility, low query performance, and absence of atomicity during writes.
Currently, the following binary files with a fixed structure are used for storing data about sessions and authentication attempts in Linux:
- /var/log/lastlog — время последнего входа (структура «struct lastlog» с полем «ll_time» 32-разрядного типа time_t);
- /var/log/btmp — неудачные попытки входа;
- /var/run/utmp — текущие сеансы;
- /var/log/wtmp — история входов и выходов.
The file data format was developed several decades ago and has a number of fundamental limitations:
- The 'tv_sec' field in the 'utmpx' structure and the 'll_time' field in 'lastlog' are of type 'int32_t', and the time counters based on them will overflow on January 19, 2038. Due to ABI compatibility requirements, these fields remain 32-bit even on 64-bit systems, thus the issue will affect all Linux installations.
- The fixed size of records prevents the addition of new fields (for example, container ID, service name, IP address) without completely replacing the format and recompiling all utilities.
- The last, lastb, who, and lastlog utilities are forced to linearly iterate through the contents of the files. With large log sizes and without the use of indexes that allow for efficient filtering of records, the input/output system load and delays in executing queries become unacceptable.
- Writing to a binary file is not an atomic operation. In the event of a failure, the write may be partially corrupted.
- To avoid conflicts when multiple processes (for example, sshd and login) write to the log simultaneously, flock locks are used, which do not guarantee atomicity and can lead to deadlocks.
The RFC author suggests completely abandoning binary formats in favor of specialized shared libraries using SQLite. A separate library is created for each type of log, with a uniform C interface: liblastlog2, libbtmp2, libutmp2, and libwtmp2. All libraries work with a database schema that includes 64-bit timestamps (INTEGER type) and indexes on users and time. There is the possibility of adding new fields without breaking compatibility (via ALTER TABLE).
Among the arguments in favor of using SQLite is the use of a 64-bit INTEGER type for storing epoch time, utilizing indexes to reduce I/O by selectively accessing records instead of full scans, the ability to add new fields without changing existing records, support for ACID transactions, and WAL (Write-Ahead Logging) mode for concurrent access without locks, as well as the proven reliability of SQLite.
To ensure a smooth transition, a dual-write strategy is proposed:
- Programs that write to binary files (login, sshd, sudo, cron, etc.) are modified to simultaneously write to both the old binary file and the new SQLite database through the corresponding library.
- New versions of utilities (last2, lastb2, who2, lastlog2) are being developed to read data from SQLite databases, using indexes for fast operations. The old utilities continue to work with the previous files.
- In a few years, when the vast majority of systems are updated, support for writing to old formats may be turned off, and the old utilities declared obsolete.
Questions raised for further discussion:
- The feasibility of splitting into separate libraries or combining into one (e.g., libsession2).
- Choice of names for libraries and utilities (whether to keep historical names or switch to more generic ones).
- Location of database files (/var/lib/ like for application state or /var/log/ like for logs).
- Mechanism for schema versioning and migration.
- SQLite performance parameters for various scenarios (servers, embedded systems).
- Providing a fallback backend that stores logs in a simplified binary format for systems where SQLite may be excessive (e.g., embedded devices with strict memory constraints).
Source: opennet.ru
