The InterSystems IRIS DBMS supports intriguing structures for data storage—globals. Essentially, these are multi-level keys with various additional features such as transactions, fast functions for traversing data trees, locking mechanisms, and its own ObjectScript language.
Learn more about globals in the series of articles 'Globals — the Sword of Data Storage':
I became curious about how transactions are implemented in globals and what specific features they have. After all, this structure for data storage is fundamentally different from the familiar tables. It's much more low-level.
As known from relational database theory, a good implementation of transactions must meet the following requirements: :
A — Atomicity. All changes made in a transaction must be recorded, or none at all.
C — Consistency. After a transaction is completed, the logical state of the database must be internally consistent. This requirement largely concerns the programmer, but for SQL databases, it also involves foreign keys.
I — Isolation. Transactions running in parallel must not affect each other.
D — Durability. After a transaction is successfully completed, issues at lower levels (like power failures) must not affect the data modified by the transaction.
Globals are non-relational data structures. They were created for ultra-fast operation on very limited hardware. Let’s take a closer look at the implementation of transactions in globals using .
To support transactions in IRIS, the following commands are used: , , .
1. Atomicity
It’s easiest to check atomicity. Let’s check from the database console.
Kill ^a
TSTART
Set ^a(1) = 1
Set ^a(2) = 2
Set ^a(3) = 3
TCOMMITThen we output:
Write ^a(1), “ ”, ^a(2), “ ”, ^a(3)We will get:
1 2 3Everything is in order. Atomicity is preserved: all changes have been recorded.
Let's complicate the task, introduce an error, and see how the transaction is preserved—partially or not at all.
Let’s check atomicity once more:
Kill ^A
TSTART
Set ^a(1) = 1
Set ^a(2) = 2
Set ^a(3) = 3After which we will forcibly stop the container, start it again, and check.
docker kill my-irisThis command is practically equivalent to a forced power off, as it sends an immediate process termination signal SIGKILL.
Could the transaction have been saved partially?
WRITE ^a(1), ^a(2), ^a(3)
^
^a(1)— No, it wasn't saved.
Let's test the rollback command:
Kill ^A
TSTART
Set ^a(1) = 1
Set ^a(2) = 2
Set ^a(3) = 3
TROLLBACK
WRITE ^a(1), ^a(2), ^a(3)
^
^a(1)Nothing was saved either.
2. Consistency
Since in global databases keys are also created on globals (reminding that a global is a lower-level data storage structure than a relational table), to fulfill the consistency requirement the key change must be included in the same transaction as the global change.
For example, we have a global ^person, where we store personal data and use the tax ID as the key.
^person(1234567, 'firstname') = 'Sergey'
^person(1234567, 'lastname') = 'Kamenev'
^person(1234567, 'phone') = '+74995555555
...In order to facilitate quick searching by surname and first name, we created the key ^index.
^index('Kamenev', 'Sergey', 1234567) = 1To keep the database consistent, we must add the individual as follows:
TSTART
^person(1234567, 'firstname') = 'Sergey'
^person(1234567, 'lastname') = 'Kamenev'
^person(1234567, 'phone') = '+74995555555
^index('Kamenev', 'Sergey', 1234567) = 1
TCOMMITAccordingly, when deleting we must also use a transaction:
TSTART
Kill ^person(1234567)
ZKill ^index('Kamenev', 'Sergey', 1234567)
TCOMMITIn other words, fulfilling the consistency requirement lies entirely on the shoulders of the programmer. But when it comes to globals — this is normal, given their low-level nature.
3. Isolation
This is where things get complicated. Many users are simultaneously working on the same database, modifying the same data.
The situation is comparable to when many users are simultaneously working with the same code repository and trying to commit changes to many files at once.
The database must handle all of this in real-time. Given that in serious companies there is even a dedicated person responsible for version control (for merging branches, resolving conflicts, etc.), and the database must do all this in real-time, the complexity of the task and the correctness of the database design and the code that supports it become apparent.
The database cannot understand the intent of the actions performed by users to prevent conflicts when they are working on the same data. It can only roll back one transaction that contradicts another or execute them sequentially.
Another problem is that during the execution of a transaction (before the commit), the state of the database may be inconsistent. Therefore, it’s preferable for other transactions to have no access to this inconsistent state, which is achieved in relational databases in various ways: by creating snapshots, using multi-versioning of rows, etc.
When executing transactions in parallel, it is important that they do not interfere with each other. This is the property of isolation.
SQL defines 4 levels of isolation:
- READ UNCOMMITTED
- READ COMMITTED
- REPEATABLE READ
- SERIALIZABLE
Let's consider each level separately. The costs to implement each level increase almost exponentially.
READ UNCOMMITTED — this is the lowest level of isolation, but it is also the fastest. Transactions can read changes made by each other.
READ COMMITTED — this is the next level of isolation, which is a compromise. Transactions cannot read changes made by each other until after the commit, but they can read any changes made after the commit.
If we have a long transaction T1 during which commits occurred in transactions T2, T3 … Tn that worked with the same data as T1, then when querying data in T1, we will receive different results each time. This phenomenon is called non-repeatable read.
REPEATABLE READ — at this level of isolation, we do not experience the non-repeatable read phenomenon because a snapshot of the data results is created for each read request, and the data from the snapshot is used in subsequent reads within the same transaction. However, at this isolation level, reading phantom data is possible. This refers to reading new rows that were added by parallel committed transactions.
SERIALIZABLE — the highest level of isolation. It is characterized by the fact that data being used in some way in the transaction (reading or modifying) becomes available to other transactions only after the first transaction is completed.
First, let's determine whether there is isolation of operations in the transaction from the main thread. We'll open 2 terminal windows.
Kill ^t
Write ^t(1)
2
TSTART
Set ^t(1)=2There is no isolation. One thread can see what the other thread doing and that opened the transaction.
Let's see if transactions from different threads can perceive what happens inside them.
We'll open 2 terminal windows and initiate 2 transactions simultaneously.
kill ^t
TSTART
Write ^t(1)
3
TSTART
Set ^t(1)=3
Parallel transactions can see each other's data. Thus, we have achieved the simplest yet fastest isolation level, READ UNCOMMITTED.
This was somewhat expected for globals, where performance has always been a priority.
What should we do if we need a higher level of isolation in operations on globals?
Here, we need to think about why isolation levels are necessary at all and how they function.
The highest level of isolation, SERIALIZE, ensures that the outcome of concurrently running transactions is equivalent to their sequential execution, thus guaranteeing the absence of collisions.
We can accomplish this using effective locks in ObjectScript, which have a variety of applications: we can perform regular, incremental, and multiple locks using the command .
Lower isolation levels are compromises aimed at increasing database performance.
Let's look at how we can achieve different levels of isolation using locks.
This operator allows for not only exclusive locks necessary for changing data, but also so-called shared locks, which can be held simultaneously by multiple threads when they need to read data that should not be changed by other processes while reading.
More on the two-phase locking method in Russian and English:
→
→
The challenge is that during a transaction, the state of the database can be inconsistent, but this inconsistent data is visible to other processes. How can we avoid this?
We will create visibility windows with locks where the state of the database will be consistent. All accesses to these visibility windows of consistent state will be controlled by locks.
Shared locks on the same data are reusable — multiple processes can acquire them. These locks prevent other processes from modifying the data, meaning they are used to establish windows of consistent database state.
Exclusive locks are used for data modifications — only one process can acquire such a lock. An exclusive lock can be acquired by:
- Any process if the data is free
- Only the process that has a shared lock on the data and was the first to request an exclusive lock.
The narrower the visibility window, the longer other processes have to wait, but the more consistent the database state can be within it.
READ_COMMITTED — the essence of this level is that we only see committed data from other threads. If the data in another transaction has not yet been committed, we see its old version.
This allows us to parallelize work instead of waiting for a lock to be released.
Without special tricks, we will not be able to see the old version of data in IRIS, so we will have to rely on locks.
Accordingly, we will need to allow data reading only at moments of consistency using shared locks.
Let's say we have a user database ^person, where users transfer money to each other.
The moment of transfer from person 123 to person 242:
LOCK +^person(123), +^person(242)
Set ^person(123, amount) = ^person(123, amount) - amount
Set ^person(242, amount) = ^person(242, amount) + amount
LOCK -^person(123), -^person(242)The moment of querying the amount of money from person 123 before withdrawal must be accompanied by an exclusive lock (by default):
LOCK +^person(123)
Write ^person(123)If we need to display the account status in the personal dashboard, we can use a shared lock or not use it at all:
LOCK +^person(123)#”S”
Write ^person(123)However, if we assume that database operations are performed almost instantaneously (let me remind you that globals are a much lower-level structure than a relational table), then the necessity for this level diminishes.
REPEATABLE READ — in this level of isolation, it is allowed that there may be multiple reads of data that can be modified by parallel transactions.
Accordingly, we will need to place a shared lock on the data we are reading and exclusive locks on the data we are modifying.
The LOCK operator allows you to detail all the necessary locks in a single operator, which can be quite numerous.
LOCK +^person(123, amount)#”S”
reading ^person(123, amount)other operations (during this time, parallel threads are trying to change ^person(123, amount), but cannot)
LOCK +^person(123, amount)
modifying ^person(123, amount)
LOCK -^person(123, amount)
reading ^person(123, amount)
LOCK -^person(123, amount)#”S”When listing locks separated by commas, they are taken sequentially, but if done like this:
LOCK +(^person(123),^person(242))they are taken atomically all at once.
SERIALIZE — we will need to set locks in such a way that ultimately all transactions that share data execute sequentially. For this approach, most locks need to be exclusive and taken on the smallest areas of the global for performance.
If we talk about funds withdrawal in global ^person, then only the SERIALIZE isolation level is acceptable, since funds must be spent strictly in sequence, otherwise it is possible to spend the same amount multiple times.
4. Durability
I conducted tests with the hard shutdown of the container via
docker kill my-irisThe database handled it well. No issues were detected.
Conclusion
For globals in InterSystems IRIS, there is support for transactions. They are truly atomic, reliable. However, to ensure database consistency in globals requires programmer effort and the use of transactions, as there are no complex built-in constructs like foreign keys.
The isolation level for globals without using locks is READ UNCOMMITTED, and with locks, it can be ensured up to the SERIALIZE level.
The correctness and speed of transactions on globals heavily depend on the programmer's skill: the more widely shared locks are used during reading, the higher the isolation level, while the more narrowly exclusive locks are taken, the better the performance.
Source: habr.com
