So, your team has finished the alpha version of your blockchain, and it's time to launch testnet, and then mainnet. You have a real blockchain, with independent participants, a good economic model, security, you have designed governance and now it's time to try it all in action. In an ideal cryptoanarchic world, you put the genesis block on the network, the final code of the node and the validators themselves start everything, raise all the auxiliary services and everything happens by itself. But this is in a fictional world, and in the real world, the team must prepare quite a lot of auxiliary software and various manipulations to help validators launch a stable network. This article is about this.
Launching networks based on “proof-of-stake” consensus, where validators are determined by the votes of system token holders, is a rather specific event, because even launching traditional, centrally managed systems with dozens and hundreds of servers is not an easy task in itself, and blockchain needs to be started with efforts loyal but independent participants. And, if in a corporation, at startup, administrators have full access to all machines, logs, and general monitoring, then validators will not let anyone into their servers and, most likely, will prefer to build their infrastructure on their own, because it controls access to the main assets of the validator - stakes voting. It is this behavior that makes it possible to build distributed secure networks - the independence of the cloud providers used, virtual and “baremetal” servers, different operating systems, all this makes attacks on such a network extremely inefficient - too much different software is used. For example, Ethereum uses two main node implementations, Go and Rust, and an attack that is effective for one implementation does not work for another.
Therefore, all the processes of launching and operating blockchains should be organized so that any validator, or even a small group of validators, could throw their computers out the window at any time and leave, while nothing should break and the remaining validators should continue to effectively support the work. networks and connect new validators. When the network is launched, when one validator is in Europe, the second in South America, and the third in Asia, it is rather difficult to achieve coordinated work of several dozen independent groups and get them interested as a result.
Validators
Let's imagine the launch of a hypothetical modern blockchain (most of what is described is suitable for blockchains based on any modern family of blockchains: Ethereum, EOS, Polkadot, Cosmos and others that provide proof-of-stake consensus. The main actors of such blockchains are the validating teams who install their own independent servers that validate and produce new blocks, and receive rewards provided by the network for those who participate in the consensus.To launch new networks, several dozen validators are required (this number can now more or less effectively reach consensus in seconds), so the project announces registration, in which validators share public information about themselves with users, convincing them that they are going to provide quality service to the launched network.
Validation is a business that allows you to extremely accurately assess the potential income of the validator, quickly transfer capacities between projects, and if the network chosen by him is successful, the validator can develop the project as a full member of the DAO and the responsible person, or simply provide an excellent technical service for completely transparent, honestly earned money. When calculating the reward for validators, projects try to take into account the costs of validators and make the block reward such that this business is profitable, but at the same time it would not allow validators to bring down the economy by flooding them with money and depriving them of the rest of the network users.
The business of validators requires high fault tolerance of services, which means a high level of training of devops and developers and expensive computing resources. Even without the need to mine hashes in proof-of-work networks, a blockchain node is a large service that takes up a lot of memory, consumes a lot of calculations, validates, writes to disk, and sends large amounts of data to the network. To store the transaction log and block chains for a blockchain with several thousand small transactions in a block, storage of 50 Gb or more is now required, and for blocks it must be an SSD. The state database of blockchains with support for smart contracts can already exceed 64Gb of RAM. Servers with the required characteristics are quite expensive, an Ethereum or EOS node can cost from 100 to 200 $/month. Add to this the increased pay for round-the-clock work of developers and devops, who during the launch period solve problems even at night, since some of the validators can easily be located in another hemisphere. However, in good times, owning a validator node can bring in serious income (in the case of EOS, up to $10 per day).
Validation is just one of the new potential IT roles for entrepreneurs and companies, as programmers come up with more and more sophisticated algorithms that allow rewarding honesty and punishing deceit and theft, services appear that perform the functions of publishing important data (oracles), performing supervision (deposit slashing and punishing cheaters by publishing proof of cheating), dispute resolution, insurance and options services, even garbage collection is a potentially big market in smart contract systems where you have to pay for data storage.
Blockchain Launch Issues
The openness of the blockchain, which made it possible to freely participate in the network of computers from any country and the ease of connecting any script kiddie to the network according to the instructions on GitHub, is not always an advantage. The pursuit of a new token often forces validators to “mine a new coin at the start”, in the hope of an increase in the rate and the ability to quickly throw off what they have earned. Also, this means that anyone can be your validator, even an anonymous one, you can vote for him in the same way as for other validators (although it will be difficult for an anonymous person to collect stakeholder votes for himself, so let's leave scary tales about anonymous cryptocurrencies to politicians) . Nevertheless
The project team has a task - how to get into their network those who in the future are able to ensure the stable operation of the nodes, understand security, know how to quickly solve problems, cooperate with other validators and act jointly - the quality of the very a token in which network participants are going to invest their time and resources. Adequate founders, assessing the risks, are well aware that when launching software of this size, they will definitely encounter errors in the code, node configuration, and that the stability of the network depends on how well developers and validators jointly solve such problems.
The team is ready to vote in the mainnet for any validators, but I would like to know for which ones, which ones are good? The biggest portfolio? Almost no one has it now. Team profiles on Linkedin? Experienced devops or security guards will not give you any Linkedin profiles. By statements in the chat, posts and helping others in the preparation phase? Okay, but subjective and inaccurate.
In such conditions, one thing remains - something that solves the problems of everyone well - a game in which it will be possible to choose the best validators, but the main thing is to test the blockchain for strength and conduct a full-scale battle test of the blockchain in conditions of active use, changes in consensus, the appearance and correction of errors . This procedure was first submitted as a game by the guys from the Cosmos project, and this idea is undoubtedly a great way to prepare the network for the launch of a reliable and fault-tolerant mainnet.
Game of Validators
I will describe the game of validators as we designed it for the DAO.Casino (DAOBet) blockchain based on the EOS fork called Haya and has a similar governance mechanism - validators are selected by voting from any account, in which part of the balance that votes for the validator is frozen. Any account that has the main BET token on its balance can vote for the selected validator with any part of its balance. The votes are summed up and based on the results, the top validators are built. This process is organized differently in different blockchains, and usually it is in this part that the new blockchain differs from the parent one, and I must say that in our case, EOS fully justifies the “OS” in its name, we really use EOS as the base operating system for deploying a modified version of the blockchain for DAOBet tasks.
I will describe individual problems and how they can be solved within the game. Imagine a network in which your server can be openly attacked, where in order to maintain the position of the validator, you need to continuously interact with the network, promoting your validator and making sure that it produces blocks and they are delivered to the other validators on time, otherwise, the validator will be thrown out of the list.
How to choose top winners?
The main technical requirement for a game is that its results be publicly verifiable. This means that the results of the game: TOP winners, must be formed strictly on the basis of data that any participant can check. In a centralized system, we could measure the “uptime” of each validator and reward those who were online the most or passed the maximum network traffic through them. You can collect data on the load of the processor, memory and reward those who have worked with dignity. But any such collection of metrics means the existence of a collection center, and the nodes are all independent and can behave as they want and send any data.
Therefore, the natural solution is that the winners should be determined according to the data from the blockchain, since it can be used to see which of the validators produced which block and which transactions were included in it. We called this number Validator Points (VP), and earning them is the main goal of the validators in the game. In our case, the simplest, easily publicly verifiable, and effective validator “utility” metric is VP = number_of_blocks_produced_by_validator in a given time period.
Such a simple choice is due to the fact that the governance in EOS already provides for many emerging problems, since EOS is the heir to already three generations of actually working blockchains with extensive experience in complex network management, and almost any validator problems with network, processor, disk lead to only one problem - he signs fewer blocks, gets paid less for his work, which again leads us simply to the number of blocks signed - for EOS this is a great and easy option.
For other blockchains, the way Validator Points are calculated may differ, for example, for pBFT-based consensuses (Tendermint/Cosmos, Aura consensus from Parity Substrate), where each block must be signed by multiple validators, it makes sense to count individual validator signatures, not blocks, perhaps it makes sense to take into account unfinished rounds of consensus, which waste the resources of other validators, in general, this is highly dependent on the type of consensus.
How to simulate real operating conditions
The task of the founders is to test the validators in conditions close to reality, while not having any centralized control. This problem can be solved with a faucet contract that distributes equal amounts of the main token to validators and everyone else. To receive tokens on the balance, it is necessary to form a transaction, and ensure that the network includes it in the block. Thus, in order to win, the validator needs to constantly replenish his balance with new tokens and vote for himself, promoting himself in the top. This activity creates a constant load on the network, and the parameters can be adjusted so that the request flow is severe enough for a full network test. Therefore, plan the faucet contract in advance as an important tool for launching the network and start selecting its parameters in advance.
Requesting tokens from the faucet and voting validators still do not fully emulate the operation of the BC, especially in extremely loaded modes. Therefore, the blockchain team will still have to write additional benchmarks one way or another to load the network. A special role in this is played by smart contracts specially created in advance, which allow testing a separate subsystem. To test storage, the contract stores random data in the blockchain, and to check network resources, the test contract requires a large amount of input data, thereby inflating the volume of transactions - by launching a stream of such transactions at arbitrary times, the team simultaneously tests the stability of the code and the durability of the validators.
A separate issue is updating the node code and conducting hard forks. It is required that in the event of a bug, vulnerability, collusion of malicious validators, the validators would have an action plan already worked out in the game of validators. Here you can come up with schemes for accruing VP for the quick application of a hard fork, for example, penalizing all validators who have not yet rolled out a new version of the node code, but this is difficult to implement, complicates the calculation. You can simulate the emergency use of a hard fork by artificially “breaking” the blockchain on a given block. Block production is halted and the winners will be those who get up early and start signing blocks, so VP based on the number of blocks signed is a good fit here.
How to inform participants about the state of the network and fix errors
Despite the distrust between the validators, it is beneficial for everyone to receive up-to-date information about the state of the network in a timely manner in order to make decisions faster, so the project team is raising a service for collecting and visualizing a variety of metrics from the validators' servers, which allows you to see the situation simultaneously for the entire network, allowing you to quickly determine what happening. Also, it is beneficial for both the validators and the project that the project team quickly fix the errors found, therefore, in addition to collecting metrics, it makes sense to immediately start collecting logs and error data from the machines of the validators to the machine available to the blockchain developers. It is not profitable for anyone to distort information here, so these services are raised by the project team and they can be trusted. It makes sense to collect system metrics from validators, and, of course, the most important metrics of the blockchain itself - for DAOBet - are the finalization time and the backlog of the last finalized block. Due to this, the team sees an increase in memory consumption on the nodes when running the benchmark, the problems of individual validators
Important points for the game of validators
As it turned out, if you want to officially allow validators to attack each other's machines (unofficially, they can do it anyway) - you need to separately legally formulate this as security testing, since according to the legislation of some countries, DDoS or network attacks can be punished. Another important issue is how to reward validators. The natural prizes are project tokens that will be transferred to the mainnet, but a massive distribution of tokens to anyone who was able to run a node is also not the best option. Most likely you will have to balance between two extreme options:
Distribute the entire prize pool according to earned VP
it is very democratic and allows everyone who has invested time and resources in the game of validators to earn
but attracts random people to the game without prepared infrastructure
Distribute top-N prize pool to validators at the end of the game
the winners are likely to be the validators who have held out the most consistently throughout the game and are very determined to win
some of the validators will not want to participate, estimating their chances of winning low, especially if there are venerable validators among the participants
Which option you prefer is up to you.
There is one more thing - it’s not at all a fact that dozens of validators will rush to participate in the game at your call, and of those who decide to try, not everyone will even install and run the node - usually, at this stage, projects have rather poor documentation, errors come across, and developers working in time pressure do not answer questions very quickly. Therefore, before launching the game, it is also necessary to provide for actions if the number of validators is not reached. In this case, at the start of the game, the missing validators are launched by the project team, participate in the consensus, but cannot be winners.
Conclusion
In conclusion, I tried to compile from the above a list of what needs to be thought out, done and launched in order to effectively conduct the game of validators
What you need to do to start a real game of validators:
develop your blockchain 🙂
- make and raise a web interface and provide a CLI for voting for validators
- make sure that metrics from a running validator node can be sent to a centralized service (for example, Prometheus)
- set up a metrics collection server (Prometheus + Grafana) for the game of validators
- figure out how Validator Points (VP) will be calculated
- develop a public script that calculates the VP of the validator based on data from the blockchain
- develop a web interface to display the top validators, and the status of the validators game (how much time is left to the end, who has how many VPs, etc.)
- develop and automate the launch of an arbitrary number of own nodes, design the process of connecting validators to the game (when and how to turn off your nodes, give and remove votes for them)
- calculate how many tokens to issue and develop a faucet contract
- make script benchmark (token transfers, massive storage usage, massive network usage)
- gather all participants in one chat for quick communication
- start the blockchain a little before the start of the game
- wait for the starting block, start the game
- test the network with several types of transactions
- roll hardfork
- change the list of validators
- repeat steps 13,14,15 in a different order, maintaining the stability of the network
- wait for final block, end game, calculate VP
I must say that the game of validators is a new story, and was held only a couple of times, so you should not take this text as a ready-made guide. There are no analogues in the modern IT business - imagine that banks, before launching a payment system, compete among themselves who will be the best to conduct customer transactions. Traditional approaches are unlikely to help you create large decentralized networks, so master new business models, run your games, identify deserving ones, reward them and keep your distributed systems running fast and stable.
Source: habr.com