Today, solutions for testing and benchmarking blockchains are tailored to a specific blockchain or its forks. But there are also a few more general solutions that differ in functionality: some of them are open source projects, others are provided as SaaS, but most of them are internal solutions created by the blockchain development team. However, they all solve similar problems. In this article, I tried to briefly review several products designed specifically for blockchain testing.
The operation of a blockchain network resembles that of a distributed database, so similar tools and methods can be used for testing. For a better understanding of how distributed databases are tested, take a look at a good selection of resources and articles . For example, latency is sorted on the shelves in this , and to understand how bugs are looked for in replication algorithms, I recommend that you familiarize yourself with this .
I will describe several popular blockchain testing and benchmarking solutions. I will be glad if in the comments you describe other useful software products for solving the same problems.
Iβll start with a tool, although not created specifically for blockchains, but which allows you to effectively test their work, provided that there is already a running network on which you can experiment. The most important factor in the reliability of a distributed system is the ability to continue working in the event of server and network problems. These can be network lags, disk overflows, unavailability of external services (DNS), hardware failures, and hundreds of other reasons. To check the stability of any systems running in concert on a large number of machine systems, you can use . It uses a highly efficient approach called Chaos Engineering.
Using its own network agent, Gremlin creates many different types of problems on the required number of machines: network lags, overload of any resource (CPU, disk, memory, network), disables certain protocols, etc. For blockchains, Gremlin can be used on testnet servers, simulating real problems and observing the behavior of the network. With it, developers and administrators in a controlled environment can see what happens in the event of a system failure or code update. In this case, the network must be configured and deployed in advance, as well as set up the collection of the necessary metrics.
Gremlin is a handy tool for architects, devops and security professionals and a universal solution for testing any ready-made and running distributed systems, including blockchains.
Hyperledger Caliper is a much more specialized solution . At the moment, Caliper supports several blockchains at once - representatives of the Hyperledger family (Fabric, Sawtooth, Iroha, Burrow, Besu), as well as Ethereum and the FISCO BCOS network.
Using Caliper, you can set the topology of the blockchain network and contracts for testing, as well as describe the configuration of the node. Blockchain nodes are raised in docker containers on the same machine. Next, you can select the required and get a file with a test result report after running. A complete list of Caliper metrics and benchmarking approach can be found here , this is a great article if you are interested in the topic of blockchain benchmarking. Also, you can set up the collection of metrics in a separate Prometheus / Grafana.
Hyperledger Caliper is a tool aimed at developers and systems architects, as it provides test repeatability and automation of testing and benchmarking. It is used in the development of the blockchain core: consensus algorithms, a virtual machine for processing smart contracts, a peer-to-peer layer, and other system mechanisms.
is a tool that appeared in the process of developing consensus and finality algorithms for networks based on EOS and testing parachains based on Parity Substrate (Polkadot). In terms of functionality, it is close to Hyperledger Caliper, as it allows you to collect important metrics from the nodes of any distributed system and client machines running test scripts.
MixBytes Tank uses several cloud services (Digital Ocean, Google Cloud Engine, etc.), in which it can launch many nodes, carry out preliminary configuration procedures, run several benchmarks on different machines in parallel, collect the necessary metrics and automatically shut down the network.
MixBytes Tank allows you to save money on cloud servers by automatically shutting down unnecessary resources after a test. Another distinguishing feature is the use of the Molecule package, which allows the developer to test the deployment of the desired blockchain locally.
MixBytes Tank allows early detection of bottlenecks and errors in algorithms that occur in real networks with a large number of geographically distributed servers and clients. The tank will help you understand what will happen on the nodes if clients send transactions with a given tps under highly repeatable conditions and with a real number of nodes spread across different continents, if necessary.
Whiteblock Genesis is an Ethereum based blockchain testing platform. This tool has a fairly wide functionality: it allows you to start the network, create the required number of accounts in it, raise the required number of clients, configure the network topology, specify the bandwidth, packetloss parameters and run the test.
Whiteblock Genesis provides its own testing facilities. Developers just need to specify the test parameters, run them using the ready-made API and get the results using a convenient dashboard.
Whiteblock Genesis allows you to configure a fairly detailed test that the platform will automatically run for each significant code change. This will allow you to catch errors at an early stage and immediately assess the impact of changes on important network parameters, such as the speed of transactions and the resources consumed by the nodes.
Madt
Another interesting young product for testing distributed systems is . It is written in Python and allows you to create the necessary network topology and the required number of servers and clients using a simple configuration script (). After that, the service deploys the network in several docker containers and raises a web interface in which you can observe messages from the servers and clients of the network. Madt can be used to test blockchains - the project repository has a p2p network test based on the Kademlia protocol, in which delays in the delivery of data to nodes gradually increase and the status of this data is checked.
Madt is a relatively new product, but given its very flexible architecture, it can evolve into a functional product.
Other solutions
Almost any testing of the system part of blockchains requires running preliminary scenarios, preparing accounts and conditions for the test (this can be testing consensus errors that can generate multiple chain forks, working out hard fork scenarios, changing system parameters, etc.). All these manipulations are carried out differently in different blockchains, so it is easier for teams to gradually adapt product testing and benchmarking to internal CI / CD and use their own developments, which gradually become more complicated with the development of blockchain functionality.
Nevertheless, the use of ready-made solutions can greatly reduce the time for these teams to test, therefore, I think that this software will be actively developed in the coming years.
Conclusion
Concluding this short review, I will list a few important characteristics of blockchain testing tools:
- The ability to automatically deploy a blockchain network under repeatable conditions. This factor is important when developing the system parts of blockchains: consensus algorithms, finality, system smart contracts.
- System ownership cost, consumed resources and convenience for continuous use. This factor provides the project with high quality tests for little money.
- Flexibility and ease of test configuration. This factor increases the chances of identifying system problems - less likely to miss something important.
- Customization for specific types of blockchains. Developing a solution based on an existing one can greatly improve quality and reduce time costs.
- Convenience and accessibility of the results obtained and their type (reports, metrics, graphs, logs, etc.). This is absolutely necessary if you want to track the history of the development of a product, or you need deep analytics of the behavior of the blockchain network.
Good luck with your testing and may your blockchains be fast and fault-tolerant!
Source: habr.com