In the PHP ecosystem, there are currently two connectors for working with the Tarantool server—this is the official PECL extension , written in C, and , written in PHP. I am the author of the latter.
In this article, I would like to share the results of performance testing of both libraries and show how with minimal code changes, you can achieve a 3-5 performance gain (on synthetic tests!).
What will we test?
We will test the aforementioned synchronous connectors running asynchronously, in parallel, and asynchronously in parallel. 🙂 We also don't want to touch the connectors’ code themselves. Currently, there are several extensions available that allow achieving the desired results:
- is a high-performance asynchronous framework for PHP. It is used by internet giants such as Alibaba and Baidu. Since version 4.1.0, a magical method has appeared, SwooleRuntime::enableCoroutine(), which allows 'with one line of code to convert synchronous network libraries in PHP into asynchronous ones.'
- Async was until recently a very promising extension for asynchronous work in PHP. Why 'was until recently'? Unfortunately, for an unknown reason, the author deleted the repository, and the future of the project is uncertain. We will have to use of the forks. Like Swoole, this extension allows you to easily enable asynchronous features by replacing the standard TCP and TLS streams with their asynchronous versions. This is done through the option 'async.tcp = 1«.
- is a relatively new extension from the well-known Joe Watkins, the author of libraries such as phpdbg, apcu, pthreads, pcov, and uopz. The extension provides an API for multithreaded work in PHP and is positioned as a replacement for pthreads. A significant limitation of the library is that it only works with the ZTS (Zend Thread Safe) version of PHP.
How will we test?
We will launch an instance of Tarantool with write-ahead logging disabled (wal_mode = none) and an increased network buffer (readahead = 1 * 1024 * 1024). The first option will eliminate disk operations, while the second will allow reading more requests from the operating system’s buffer, thereby minimizing the number of system calls.
For benchmarks that work with data (inserts, deletions, reads, etc.), a memtx space will be (re)created before the benchmark starts, in which the values of the primary index are generated by an ordered integer value generator (sequence).
The DDL of the space looks like this:
space = box.schema.space.create(config.space_name, {id = config.space_id, temporary = true})
space:create_index('primary', {type = 'tree', parts = {1, 'unsigned'}, sequence = true})
space:format({{name = 'id', type = 'unsigned'}, {name = 'name', type = 'string', is_nullable = false}})If necessary, before running the benchmark, the space is populated with 10,000 tuples of the form
{id, "tuple_"}Access to the tuples is done using a random key value.
The benchmark itself is a single request to the server that is executed 10,000 times (iterations), which are in turn executed in iterations. Iterations are repeated until all time deviations between 5 iterations are within an allowable margin of error of 3%*. After that, the average result is taken. There is a pause of 1 second between iterations to prevent the CPU from throttling. The Lua garbage collector is disabled before each iteration and forcibly started after its completion. The PHP process is launched only with the necessary extensions for the benchmark, with output buffering enabled and garbage collection disabled.
* The number of iterations, cycles, and error threshold can be changed in the benchmark settings.
Test Environment
The results published below were obtained on a MacBookPro (2015), with the operating system being Fedora 30 (kernel version 5.3.8-200.fc30.x86_64). Tarantool was running in Docker with the parameter "--network host".
Package versions:
Tarantool: 2.3.0-115-g5ba5ed37e
Docker: 19.03.3, build a872fc2f86
PHP: 7.3.11 (cli) (built: Oct 22 2019 08:11:04)
tarantool/client: 0.6.0
rybakit/msgpack: 0.6.1
ext-tarantool: 0.3.2 (+ patch for 7.3)*
ext-msgpack: 2.0.3
ext-async: 0.3.0-8c1da46
ext-swoole: 4.4.12
ext-parallel: 1.1.3
* Unfortunately, the official connector does not work with PHP versions > 7.2. To compile and run the extension on PHP 7.3, it was necessary to use .
Results
Synchronous Mode
Tarantool's protocol uses a binary format for message serialization. In the PECL connector, serialization is deeply hidden within the library, and affecting the encoding process from userland code The pure PHP connector, on the other hand, provides the ability to customize the encoding process by extending the standard encoder or by using your own implementation. Out of the box, there are two encoders available, one based on (the official MessagePack PECL extension), and the other based on (in pure PHP).
Before comparing the connectors, let's measure the performance of the MessagePack encoders for the PHP connector, and in further tests, we will use the one that shows the best result:
Although the PHP version (Pure) is slower than the PECL extension, in real projects, I would still recommend using , because the official MessagePack extension implements only part of the format specification (for example, there is no support for custom data types, without which you won't be able to use Decimal — a new data type introduced in Tarantool 2.3) and has several other (including compatibility issues with PHP 7.4). Overall, the project appears to be abandoned.
So, let's measure the performance of the connectors in synchronous mode:
As seen from the graph, the PECL connector (Tarantool) shows better performance compared to the PHP connector (Client). This is not surprising, considering that the latter, aside from being implemented in a slower language, essentially does more work: a new object is created with every call Request and Response (in the case of Select — also Criteria, and in the case of Update/Upsert — Operations), and separate entities Connection, Packer and Handler also add overhead. It is clear that flexibility comes at a cost. However, in general, the PHP interpreter shows good performance, even though there is a difference, it is insignificant and may become even smaller with the use of preloading in PHP 7.4, not to mention JIT in PHP 8.
Moving on. Tarantool 2.0 introduced support for SQL. Let's try to perform Select, Insert, Update, and Delete operations using the SQL protocol and compare the results with the noSQL (binary) equivalents:
The SQL results are not particularly impressive (remember, we are still testing synchronous mode). However, I wouldn't be too quick to be disappointed about this; SQL support is still under active development (recently, for example, support for ) was added, and judging by the list , the SQL engine will undergo a series of optimizations.
Async
Well, let's see how the Async extension can help us improve the results above. For writing asynchronous programs, the extension provides an API based on coroutines, which we will utilize. Through trial and error, we determine that the optimal number of coroutines for our environment is 25:
We will 'spread' 10,000 operations across 25 coroutines and see what we achieve:
The number of operations per second increased more than threefold for !
Unfortunately, the PECL connector did not start with ext-async.
What about SQL?
As you can see, in asynchronous mode, the difference between the binary protocol and SQL has become negligible.
Swoole
Once again, we determine the optimal number of coroutines, this time for Swoole:
We will settle on 25. We will repeat the same trick as with the Async extension—distributing 10,000 operations among 25 coroutines. Additionally, we will add another test, where we split all work into 2 processes (that is, each process will execute 5,000 operations in 25 coroutines). Processes will be created using SwooleProcess.
Results:
Swoole shows a slightly lower result compared to Async when running in a single process, but with 2 processes, the picture changes dramatically (the number 2 was not chosen randomly; on my machine, exactly 2 processes showed the best result).
By the way, the Async extension also has an API for working with processes, but there I did not notice any difference when running benchmarks in one or several processes (it’s possible that I made a mistake somewhere).
SQL vs binary protocol:
Just like with Async, the difference between binary and SQL operations is leveled in asynchronous mode.
Parallel
Since the Parallel extension is not about coroutines, but about threads, let’s measure the optimal number of parallel threads:
It equals 16 on my machine. Let's run connector benchmarks on 16 parallel threads:
As you can see, the result is even better than with asynchronous extensions (except for Swoole running on 2 processes). Note that for the PECL connector, there is nothing in the Update and Upsert operation slots. This is related to the fact that these operations failed with an error—I don't know whether it was due to ext-parallel, ext-tarantool, or both.
Now let's compare SQL performance:
Did you notice the similarity with the graph for connectors run synchronously?
All together.
Finally, let's consolidate all the results into one graph to see the overall picture for the tested extensions. We will add just one new test to the graph that we haven't done yet—running Async coroutines in parallel using Parallel*. The idea of integrating the aforementioned extensions has already by the authors, but a consensus has not yet been reached, so we’ll have to do it ourselves.
* It was not possible to run Swoole coroutines with Parallel, apparently these extensions are incompatible.
So, the final results are:
In conclusion
I believe the results are quite impressive, and for some reason, I'm confident that this is not the limit! Whether you need this in a real project is entirely up to you; I'll only say that for me, it was an interesting experiment that allowed assessing how much can be 'squeezed' out of a synchronous TCP connector with minimal effort. If you have ideas to improve the benchmarks—I would gladly consider your pull request. All the code with instructions for running and results is published separately .
Source: habr.com
