A common question from a non-technical specialist about any distributed system is, 'How many TPS does your blockchain have?'. However, the number given in response often has little to do with what the inquirer really wants to know. In reality, they want to ask, 'Will your blockchain meet my business requirements?', and these requirements are not a single number, but a multitude of conditions—network fault tolerance, finality requirements, sizes, nature of transactions, and many other parameters. So the answer to the question 'how many TPS' is unlikely to be simple and almost never complete. A distributed system with dozens and hundreds of nodes performing quite complex computations can exist in an enormous number of different states related to the state of the network, the contents of the blockchain, technical failures, economic issues, network attacks, and many other reasons. The stages at which performance problems may arise differ from traditional services, and a server in a blockchain network acts as a network service that combines the functionalities of a database, web server, and torrent client, making it extremely complex in terms of load profiles across all subsystems: CPU, memory, network, storage.

It so happens that decentralized networks and blockchains are quite specific and unfamiliar software for developers of centralized software. Therefore, I would like to highlight important aspects of performance and resilience of decentralized networks, approaches to measuring them, and identifying bottlenecks. We will examine various performance issues that limit the speed of service delivery to blockchain users and note the characteristics specific to this type of software.

Stages of service request by a blockchain client

To speak honestly about the quality of any somewhat complex service, one must consider not only the average values but also the maximum/minimum values, medians, and percentiles. Theoretically, one can talk about 1000 tps in a blockchain, but if 900 transactions were completed at great speed, while 100 'hung' for a few seconds, the average time collected across all transactions is not quite an honest metric for a client who could not complete a deal in a few seconds. Temporary 'pits' caused by missed consensus rounds or network splits can severely impair a service that showed excellent performance on test beds.

To identify such bottlenecks, it's essential to understand the stages where a real blockchain might face difficulties in serving its users. Let's describe the delivery cycle and transaction processing, as well as obtaining the new state of the blockchain, from which the client can verify that their transaction has been processed and recorded.

The transaction is formed on the client side.
The transaction is signed on the client side.
The client selects one of the nodes and sends their transaction to it.
The client subscribes to updates from the state database of the node, awaiting the results of their transaction execution.
The node propagates the transaction through the p2p network.
One or more block producers process the accumulated transactions, updating the state database.
The block producer forms a new block by processing the required number of transactions.
The block producer propagates the new block through the p2p network.
The new block is delivered to the node that the client is querying.
The node updates the state database.
The node sees the update related to the client and sends them a notification about the transaction.

Now let's take a closer look at these stages and describe the potential performance issues at each step. Unlike centralized systems, we will also consider code execution on the network clients. Quite often, when measuring TPS, the transaction processing time is collected from nodes rather than from the client — this is not entirely fair. The client doesn’t care how quickly the node processed their transaction; what matters most to them is when reliable information about that transaction, included in the blockchain, becomes available to them. This metric is essentially the transaction execution time. This means that different clients, even when sending the same transaction, can receive completely different times, depending on the channel, load, and proximity of the node, etc. Therefore, it is crucial to measure this time on the clients, as this is the parameter that needs to be optimized.

Transaction Preparation on the Client Side

Let's start with the first two points: the transaction is formed and signed by the client. Strangely enough, this can also be a performance bottleneck of the blockchain from the client's perspective. This is unusual for centralized services, where all calculations and data operations are handled internally, and the client simply prepares a short request capable of querying a large volume of data or computations, receiving a ready result. In blockchains, the client code is becoming increasingly powerful, while the blockchain core is becoming lighter, and massive computational tasks are typically delegated to client software. In blockchains, there are clients that can prepare a single transaction for quite a while (I am talking about various Merkle proofs, succinct proofs, threshold signatures, and other complex operations on the client side). A good example of light on-chain verification and heavy transaction preparation on the client is the proof of membership in a list based on a Merkle tree. article.

It is also important to remember that the client code does not just send transactions to the blockchain, but first queries the state of the blockchain—this activity can affect the load on the network and blockchain nodes. Therefore, when conducting measurements, it is wise to emulate the behavior of the client code as fully as possible. Even if your blockchain has regular lightweight clients that simply sign a basic transaction transferring some asset, the amount of computational load on the client is increasing each year, cryptographic algorithms are strengthening, and this part of the processing could become a significant bottleneck in the future. So be cautious and do not overlook a situation where, in a transaction lasting 3.5 seconds, 2.5 seconds are spent on preparation and signing, and 1.0 second is for sending to the network and awaiting a response. To assess the risks of this bottleneck emerging, metrics should be collected from client machines and not just from blockchain nodes.

Sending a transaction and monitoring its status

The next step is to send the transaction to the selected blockchain node and obtain its acceptance status into the transaction pool. This stage is similar to a regular database call, where the node needs to record the transaction in the pool and begin spreading information about it through the P2P network. The approach to assessing performance here resembles that of evaluating traditional Web API microservices, where transactions in blockchains can be updated and actively change status. In general, updating transaction information in some blockchains can occur multiple times, for instance, when switching between forks of the chain or when block producers inform about their intention to include a transaction in a block. Restrictions on the size of this pool and the number of transactions within it can influence blockchain performance. If the transaction pool is filled to its maximum capacity or cannot fit into memory—network performance can sharply decline. Blockchains do not have centralized means to protect against the flow of spam messages, and if a blockchain supports large transactions and low fees, this can lead to the transaction pool being overwhelmed—this is yet another potential performance bottleneck.

In blockchains, the client sends a transaction to any node of the blockchain that they prefer. The hash of the transaction is usually known to the client even before sending, so all they need to do is establish a connection and then wait for the blockchain to change its state by including their transaction. Note that measuring "tps" can yield completely different results depending on the connection methods to the blockchain node. This can be a standard HTTP RPC or WebSocket, which allows for the "subscribe" pattern. In the latter case, the client receives notifications earlier, and the node spends fewer resources (mainly memory and traffic) responding about the transaction's status. Therefore, when measuring "tps", it's essential to consider the client's connection method to the nodes. Thus, to assess the risks of this bottleneck, the blockchain benchmark must be capable of emulating clients with both WebSocket and HTTP RPC requests, in proportions corresponding to real networks, and also vary the nature and size of transactions.

To assess the risks of this bottleneck, metrics should also be collected from client machines, not just from blockchain nodes.

Transmission of transactions and blocks over a p2p network

In blockchains, peer-to-peer (p2p) networking is used for transmitting transactions and blocks between participants. Transactions spread across the network, starting from one of the nodes, until they reach the peers, the block producers, who package the transactions into blocks and distribute new blocks across all nodes in the network through the same p2p method. The foundation of most modern p2p networks is various modifications of the Kademlia protocol. Here a good brief overview of this protocol, and here — the article includes various measurements in the BitTorrent network, which demonstrates that such networks are more complex and less predictable than a rigidly configured centralized service network. Also, here an article on measuring various interesting metrics for Ethereum nodes.

In short, each peer in such networks maintains its own dynamic list of other peers from which it requests blocks of information addressed by content. Upon receiving a request, the peer either provides the requested information or forwards the request to the next pseudo-random peer from the list, and after receiving a response, it conveys it to the requester while caching it for a while, supplying this block of information earlier next time. Thus, popular information ends up in many caches across numerous peers, while unpopular content is gradually displaced. Peers keep track of who has transferred how much information to whom, and the network seeks to incentivize active distributors by enhancing their ratings and providing them with a higher service level, automatically expelling inactive participants from the peer lists.

Now, the transaction needs to be propagated across the network so that block producers can see it and include it in a block. The node actively "distributes" the new transaction to all interested parties while listening to the network, waiting for a block in which the required transaction will appear to notify the waiting client. The time it takes for the network to relay information about new transactions and blocks in P2P networks depends on a multitude of factors: the number of honest, operational nodes nearby (from a network perspective), how well the caches of those nodes are warmed up, the sizes of the blocks, transactions, the nature of the changes, the geography of the network, the number of nodes, and many more factors. Comprehensive measurements of performance metrics in such networks are complex, requiring simultaneous assessment of request processing times both on clients and on peers (blockchain nodes). Issues in any of the P2P mechanisms, incorrect eviction and caching of data, inefficient management of active peer lists, and many other factors can contribute to delays that affect the efficiency of the entire network as a whole, and this bottleneck is the most challenging to analyze, test, and interpret results.

Processing the blockchain and updating the state database

The most critical part of blockchain functionality is the consensus algorithm, its application to new blocks obtained from the network, and the processing of transactions with the results recorded in the state database. Adding a new block to the chain and the subsequent selection of the main chain should occur as quickly as possible. However, in real life, 'should' does not mean 'works,' and one can envision a situation where two long competing chains are constantly switching between each other, altering the metadata of thousands of transactions in the pool with each switch, and continuously reverting the state of the state database. This stage, in terms of identifying the bottleneck, is simpler than the network p2p layer, since the execution of transactions and the consensus algorithm are strictly deterministic, making measurement easier here.
The main thing is not to confuse random performance degradation at this stage with network issues—nodes are delivering blocks and information about the main chain more slowly, which may appear to external clients as a slow network, even though the problem lies entirely elsewhere.

To optimize performance at this stage, it is useful to collect and monitor metrics from the nodes themselves, including those related to updating the state database: the number of blocks processed at the node, their size, the number of transactions, the count of switches between chain forks, the number of invalid blocks, the runtime of the virtual machine, the data confirmation time, etc. This will help distinguish network issues from errors in the chain processing algorithms.

The virtual machine processing transactions can be a valuable source of information capable of optimizing blockchain performance. The number of memory allocations, the count of read/write instructions, and other metrics related to the efficiency of executing contract code can provide developers with a wealth of useful information. At the same time, since smart contracts are programs, in theory, they can consume any resources: CPU/memory/network/storage, making transaction processing a relatively ambiguous stage that varies significantly between versions and changes to contract code. Therefore, metrics regarding transaction processing are also necessary for effective optimization of blockchain performance.

Client Notification of Transaction Inclusion in Blockchain

This is the final stage of the client receiving blockchain service. Compared to other stages, there are no major overhead costs here, but one must still consider the possibility of receiving a large response from the node (for example, a smart contract returning an array of data). In any case, this moment is crucial for those who asked, "What is the TPS in your blockchain?" as this is when the service time is recorded.

At this point, there must be an indication of the total time the client spent waiting for a response from the blockchain. This is the time the user expects confirmation in their application, and optimizing this time is the primary task for developers.

Conclusion

As a result, we can describe the types of operations performed in blockchains and categorize them into several groups:

cryptographic transformations, proof construction
peer-to-peer networking, transaction and block replication
transaction processing, smart contract execution
applying changes in the blockchain to the state database, updating transaction and block data
read-only queries to the state database, blockchain node APIs, subscription services

Overall, the technical requirements for nodes in modern blockchains are extremely demanding—fast CPUs for cryptography, large volumes of RAM for storing and quickly accessing the state database, network interactions requiring many simultaneously open connections, and substantial storage. Such high demands and the abundance of various types of operations inevitably lead to nodes potentially running out of resources, making any of the aforementioned stages a bottleneck for overall network performance.

When developing and assessing the performance of blockchains, you need to consider all these aspects. This requires collecting and analyzing metrics concurrently from clients and network nodes, searching for correlations between them, evaluating the response time to customers, taking into account all key resources: cpu/memory/network/storage, and understanding how they are utilized and influence each other. All this makes comparing the speeds of different blockchains in terms of "how many TPS" a thankless task, as there are a vast number of different configurations and states. In large centralized systems, clusters of hundreds of servers, these issues are also complex and require collecting a large number of different metrics; however, in blockchains, due to p2p networks, virtual machines, processing contracts, and internal economics, the degree of freedom is much greater, making tests even on a few servers unrepresentative and providing only very rough values that have little connection to reality.

Therefore, when developing within the blockchain core, to assess performance and answer the question "has it improved compared to last time?" we use quite sophisticated software that orchestrates the launch of the blockchain with dozens of nodes and automates benchmarking and metric collection. Without this information, it is extremely difficult to debug protocols that involve multiple participants.

So, when posed with the question "how many TPS does your blockchain have?" offer your interlocutor some tea and clarify whether they are ready to review a dozen graphs and listen to all three boxes of performance issues in blockchains along with your proposed solutions...

Source: habr.com