Lisk Mainnet Regenesis
If you have been following our research blog posts and roadmap, you will already be aware of the large number of improvements that are part of the Lisk SDK 5.0 and the next mainnet release Lisk Core 3.0.
All these improvements imply a significant change of the Lisk protocol, and, in particular, a major change in the format for accounts, transactions and blocks. You may be wondering how we perform the migration of the Lisk Mainnet to this very different protocol. Hence, the purpose of this blog post is to explain our regenesis approach to tackle the complexity of the migration.
If you have been following our research blog posts and roadmap, you will already be aware of the large number of improvements that are part of the Lisk SDK 5.0 and the next mainnet release Lisk Core 3.0. We have covered most of these major improvements in our research blog post series, and a brief overview of them is given below:
- The Lisk-BFT consensus algorithm improves the consensus algorithm by adding block finality.
- The new DPoS system makes the delegate selection more fair and open.
- The new address system improves security and allows the detection of typos in addresses.
- The new dynamic fee system greatly reduces transaction fees.
- The new serialization of blocks and transactions increases transmission and storage efficiency.
- Merkle trees used to organize transactions in blocks offer the possibility for efficient inclusion proofs for transactions.
- The new multisignature accounts allow much more flexible signing rules.
All these improvements imply a significant change of the Lisk protocol, and, in particular, a major change in the format for accounts, transactions and blocks. You may be wondering how we perform the migration of the Lisk Mainnet to this very different protocol. Hence, the purpose of this blog post is to explain our regenesis approach to tackle the complexity of the migration. For instance, you will learn how your account is migrated and how you will be able to vote in the new DPoS system. Additionally, we explain the new genesis block format that is part of Lisk SDK 5.0 and used for the regenesis approach. The full technical specifications of the new genesis block format is given in LIP 34 and the specifications of the migrations are contained in LIP 35.
In the first section of this blog post, we begin by providing some context and delve into the theoretical background. In the next section, we cover the new genesis block format in the Lisk SDK and discuss the benefits it provides for SDK developers. Finally, in the last section, we explain the regenesis approach for the Lisk Mainnet migration that is based on creating a snapshot of the Lisk blockchain and utilizing the new genesis block format.
Blockchain as a Replicated State Machine
A state machine is an abstract model for a system that can be in exactly one of a finite number of states and, depending on some inputs, changes from one state to another. Many real world machines such as traffic lights, elevators or a coffee machine can be modelled as a state machine.
A Lisk node can also be modeled as a state machine. In this case, one state of the Lisk node corresponds to the set of accounts states (addresses with associated balance, votes, delegate registration, …), and some auxiliary information such as the current height and last block ID. The overall number of states in this case is incredibly large, for example, a different balance for just one account corresponds to a different state of the whole blockchain. The inputs that cause the Lisk node to change from one state to another state are blocks. Every time the node applies a block and the transactions in the payload of the block, it transitions from one state to another state. Let us consider the concrete example below:
Figure 1: Example of a Lisk node transitioning from one state to another when applying a block.
The Lisk node is in a state, which we call s1, where a user A has 100 LSK, a user B 50 LSK and a delegate C has 600 LSK. In the block that is then applied, there is one token transfer of 20 LSK from user A to user B and the forging delegate C receives 1 LSK block reward (for simplicity we assume that there are no fees).
This means that applying the block causes the Lisk node to change to a new state, which we call s2, where user A has 80 LSK, user B 70 LSK and delegate C 601 LSK. This way, one Lisk node can be modeled as a state machine that starts with a pre-defined initial state and then changes to different states depending on which blocks are applied or removed.
The Lisk Mainnet is not composed of only one node, but hundreds of nodes that are all supposed to eventually apply the same blocks (and thus reach the same state). This is where the consensus protocol comes in. The consensus protocol is an algorithm for nodes to agree on the same block at every height. A blockchain is therefore also referred to as a replicated state machine because every node is a state machine and the consensus protocol ensures that the same states are replicated throughout the network.
State and History
A state comprises all data that is necessary to validate and apply future blocks. It is important to understand that the state at a certain height is stored by every node and can be computed by applying blocks and transactions up to that height, but is usually not explicitly part of the blockchain. For instance, a token transfer transaction with the transferred amount is included in a block, but the updated account balances are not added to the block although they are stored by every node.
Figure 2: Blockchain history and state.
For the Lisk blockchain, the state contains all account information, that is, all addresses that were ever used, their balance, nonce, votes, and possibly delegate and multisignature registration information.
Additionally, the current height, last block ID and timestamp are part of the state because they are necessary to process the next block (the next block must include the previous block ID, have a height of one larger, and a larger timestamp than the previous block). Strictly speaking some additional information related to the consensus, such as the random seeds used to select standby delegates, is also part of the state.
However, past token transfer transactions are not part of the state, for instance. If you hold 100 LSK in your account, then it does not matter if you received 1 transfer of 100 LSK or 100 transfers of 1 LSK. All that matters to process any future actions that you want to perform with your account is that you have 100 LSK.
In general, all blocks and transactions are considered to be part of the blockchain history and are not part of the state. When we say that a blockchain is immutable, we actually mean that the history of finalized blocks and transactions is immutable. In contrast to that, the state of the blockchain is constantly evolving as new blocks are applied.
New Lisk SDK Genesis Block Format
The purpose of the genesis block is to yield the desired initial state of a blockchain. An important part of the initial state is the initial distribution of tokens. Let us look at the Lisk Mainnet genesis block as an example. In order to achieve the desired token distribution, the genesis block contains token transfer transactions to all ICO participants.
These token transfers originate from a special genesis account (Lisk address 6566229458323231555L in the old format), which is allowed to have a negative balance. For a DPoS blockchain, it is also essential to define some initial delegates which can forge blocks building on the genesis block and get the whole blockchain started.
In the Lisk Mainnet genesis block this is achieved by including 101 delegate registration transactions for the delegates genesis_1,..., genesis_101, and some vote delegate transactions that vote for these delegates. The idea is that these initial delegates take over the job of forging blocks until enough users have registered a delegate and voted.
Overall, the Lisk Mainnet genesis block contains 8176 transactions making the genesis block rather large. Additionally, securely managing 101 delegate accounts and passphrases to start a new blockchain can be cumbersome. That is one of the reasons why we introduce a more compact and easy-to-use genesis block format in the Lisk SDK 5.0. The idea is that sidechain developers can define the desired initial state explicitly, rather than having to create and sign many transactions that yield the desired state.
Let us have a closer look how the initial state is defined in the new genesis block format. Essentially, the following three block properties, which are unique to the genesis block, determine the initial state:
- accounts: This property contains an array of accounts. Each account must include the account address and initial balance, while all other properties can simply be default values.
- initDelegates: The value provided for this property must be an array of delegate addresses that will forge during the bootstrap period, the initial rounds that allow users to register delegates and vote. These addresses must also be included in the accounts array and be accounts registered as delegates with a specified delegate name. Blockchain creators have the flexibility to provide only one or a few delegates here as long as the number is at most the round length of your blockchain.
- initRounds: This property defines the length of the bootstrap period, during which blocks are forged by the delegates given in initDelegates. Afterwards, the normal DPoS algorithm kicks-in and delegates are selected according to the current votes. For technical reasons this value has to be at least 3. The chosen value should be large enough so that the majority of users has enough time to register delegates and vote to avoid an unstable, drastically changing delegate set in the rounds after the end of the bootstrap period.
Overall, giving the initial state of the blockchain explicitly allows for a more compact genesis block and also simplifies the bootstrapping of a new blockchain as managing only one delegate account is sufficient. If you want to know more details about the new genesis block format, see LIP 34, which contains the full technical specifications.
Lisk Mainnet Hard Fork and Regenesis Process
Lisk Core 3.0 not only introduces a greatly improved, new protocol, but its release also implies a hard fork for the Lisk Mainnet, as the Lisk Core 2.x nodes cannot process any transactions or blocks following the new protocol of Lisk Core 3.0. All node operators that want to support the new protocol therefore have to upgrade their nodes.
For the mainnet hard fork and migrating to Lisk Core 3.0, we propose a new regenesis approach. The basic idea of the approach is illustrated in the diagram below. At the height of the hard fork, which we refer to as regenesis height, any Lisk Core 2.x operator can run a snapshot script that takes a snapshot of the state of the Lisk Core 2.x chain.
This snapshot includes all accounts, their balances and other account properties as well as the top 103 delegates. The snapshot of the state is exported as a special snapshot block which follows the format described in the previous section. The snapshot block is then required as input for the Lisk Core 3.0 node to start joining the Lisk Core 3.0 network.
Figure 3: Lisk Mainnet hard fork by creating a snapshot of the state.
This approach for the mainnet hard fork is as trustless and decentralized as previous hard forks, as every node can independently compute the snapshot block. Compared to previous hard forks, this upcoming hard fork also has some special characteristics. While Lisk Core 2.x nodes are fully backwards compatible, which means they can process any block from the genesis block onwards and also support previous protocol versions, Lisk Core 3.0 will not be fully backwards compatible and only understand blocks from the snapshot block onwards.
The reason for this is that it is difficult to have one codebase that supports two very different consensus algorithms, two address systems and very different formats for transactions and blocks. Cutting off the full backwards compatibility has the advantage of reducing the complexity of the migration significantly, as the snapshot block provides a simple interface between the old and new protocol. Moreover, not supporting all previous protocol versions greatly reduces the complexity of the code base and makes testing and introducing new features easier.
The only disadvantage is that to verify the correctness of the whole Lisk blockchain from the genesis block onwards, it is now necessary to run two implementations, first Lisk Core 2.x and then Lisk Core 3.0. As a node operator joining the Lisk Mainnet after the hard fork, you have the option to compute the snapshot block yourself using Lisk Core 2.x or to obtain it from some trusted source allowing for faster synchronization with the network.
In this section, we provide more details on how accounts are migrated as part of the hard fork.
Due to the new address system, all account addresses have to be migrated. For accounts that have sent at least one transaction, every Lisk Core 2.x node stores the public key as part of the account, which allows to compute the new address during the snapshot creation. An example of such an account migration is shown below. Note that Lisk Core 3.0 nodes will not store the public key anymore, as the new address is sufficiently long and storing the public key does not provide any additional security benefit.
Figure 4: Example of the migration of one account to Lisk Core 3.0. The address is changed to the new 20-byte value in the node, which is represented in Base32 in all front-end tools. Additionally, the public key is removed and a new nonce property is introduced.
Accounts that have never sent a transaction cannot be migrated to the new address system automatically, as their public keys are unknown. These accounts are also migrated to Lisk Core 3.0 with the old address and balance. In order to then use the balance in the Lisk Core 3.0 network, the account holder has to migrate the account to the new address system manually by sending a special reclaim LSK transaction that includes the necessary public key.
To avoid having to do this extra step in the future, you just have to send at least one transaction from your account, for example a transfer of 1 LSK to yourself. This is currently also recommended to fully secure your account as the current shorter addresses have a relatively low preimage resistance (see the address system blog post for details).
Lisk Core 3.0 introduces an improved multisignature system that enables the possibility to register a set of mandatory public keys, a set of optional public keys, and a minimum number of required signatures. A detailed explanation of multisignature accounts and these properties is given in the Multisignature Accounts blog post.
Every multisignature account existing in the Lisk Core 2.x chain is converted to a multisignature account with one mandatory key, the key that registered the account, the keys in the keysgroup property as optional keys, and the min value as the minimum number of required signatures. In this way, the signing conditions for multisignature accounts are preserved: The original account owner always has to sign and the set of public keys as well as number of required signatures remain unchanged.
Additionally, we also migrate second signature accounts to multisignature accounts. Recall that in Lisk Core 2.x a second signature registration transaction offers the possibility to register a second public key. Afterwards, both the original as well as the second public key are mandatory to sign a transaction.
A second signature account is migrated to a multisignature account where both public keys are mandatory and two signatures are required. Note that this only affects how these accounts are represented in the protocol. The signing conditions are exactly the same and these accounts will be represented in a similar fashion in the user interface. The main reason for migrating second signature accounts to multisignature accounts is to have a simpler and leaner protocol that still supports all the previous existing features.
Migration to new DPoS System
In the new DPoS system, users vote for a delegate by locking tokens, see the DPoS blog post for details. Every LSK token can only be used once for voting and users can overall only vote for up to ten delegates with one account. Due to these changes, the votes in the old DPoS system are invalid and are discarded as part of the hard fork.
This means that the Lisk Core 3.0 chain will start from the snapshot block without any votes, and therefore it is important for all users to re-vote after the hard fork. As shown in the diagram below, the top 103 delegates according to the old voting system are selected to start the Lisk Core 3.0 chain and forge for the first 600 rounds (approximately 7 days). After this time has expired, the new DPoS system becomes active and delegates will be selected according to the votes cast in these 600 rounds.
Figure 5: Migration to the new DPoS system.
We expect that 7 days is a sufficient time frame for a majority of active accounts to re-vote so that the set of active delegates will be sufficiently stable from that point onwards. Note that selecting delegates with the new DPoS system after one round, for instance, would be very risky and open the door for potential attacks.
This is due to the fact that it is relatively easy for an attacker to control a majority of delegates if only a few accounts actually vote, given that the attacker may also prevent others from voting in such a short timespan by submitting a large number of high fee transactions.
This blog post introduces the new Lisk SDK genesis block format, as defined in LIP 34, and explains the regenesis approach for next Lisk Mainnet migration, as fully specified in LIP 35. For Lisk users the key message here is that your funds will be migrated automatically in a secure manner.
Remember to pay attention to our social media channels for the date of the Lisk Mainnet hard fork so you can re-vote after the migration. For node operators, this blog post hopefully provides some helpful background information for the future hard fork. We will present a further detailed guide for upgrading to Lisk Core 3.0 in due course.
For additional questions and comments on the topic, we will host an AMA on Lisk.chat with Jan Hackfeld next Friday, October 23rd at 4pm CEST. We also invite all community members to go to the Lisk Research forum, where we are always happy to hear your feedback and to participate in discussions on this and other topics.