Building blockchain based on Java language (3) — persistence & command line

Time:2020-11-7

Building blockchain based on Java language (3) -- persistence & command line

The final content is subject to the original texthttps://wangwei.one/posts/35c…

introduction

LastIn this paper, we implement the proof of work mechanism (POW) of blockchain to realize mining as much as possible. However, there are still many important features that have not been implemented from the real blockchain application. Today, we will implement the storage mechanism of blockchain data and save the blockchain data generated each time. It should be noted that blockchain is essentially a distributed database. We do not implement “distributed” here, but only focus on data storage.

Database selection

So far, our block storage mechanism has not been saved every time. This is not convenient for us to reuse the blockchain. We have to generate blocks from the beginning every time, nor can we share our blockchain with others. Therefore, we need to store it on disk.

Which database should we choose? In fact, in《Bitcoin white paper》There is no explicit specification for which database to use in, so it’s up to the developer to decide.NakamotoDevelopedBitcoin CoreIs used inLevelDB。 original textBuilding Blockchain in Go. Part 3: Persistence and CLIIs used inBoltDBIt supports go language better.

However, we use Java for implementation. Boltdb does not support Java. Here we choose JavaRocksdb

Rocksdb is a key value storage engine developed and maintained by Facebook database engineering team. It has more powerful performance than leveldb. For detailed introduction of rocksdb, please refer to the official document:https://github.com/facebook/r…I’m not going to give you more.

data structure

Before we start to implement data persistence, we need to determine how we should store our data. So let’s take a look at how bitcoin works.

In short, bitcoin uses two “buckets” to store data:

  • blocks. metadata describing all blocks in the chain
  • chainstate. store the status of the blockchain, which refers to all theUTXO(no transaction output is spent) and some metadata

“In the bitcoin world, there are no accounts and no balances, only utxo dispersed in the blockchain.”

See:Mastering bitcoin, 2nd Edition, chapter 06: input and output of transactions

In addition, each chunk data is stored on disk as a separate file. This is done for performance reasons: when reading a single block data, it is not necessary to load all the chunk data into memory.

stayblocksIn this bucket, the key value pairs stored are:

  • ‘b’ + 32-byte block hash -> block index record

    Index record of the block

  • ‘f’ + 4-byte file number -> file information record

    Document information record

  • ‘l’ -> 4-byte file number: the last block file number used

    The file encoding used by the latest block

  • ‘R’ -> 1-byte boolean: whether we’re in the process of reindexing

    Is it in the process of re indexing

  • ‘F’ + 1-byte flag name length + flag name string -> 1 byte boolean: various flags that can be on or off

    Various flags that can be turned on or off

  • ‘t’ + 32-byte transaction hash -> transaction index record

    Transaction index records

staychainstateIn this bucket, the key value pairs stored are:

  • ‘c’ + 32-byte transaction hash -> unspent transaction output record for that transaction

    Utxo record of a transaction

  • ‘B’ -> 32-byte block hash: the block hash up to which the database represents the unspent transaction outputs

    The hash of utxo block represented by the database (sorry, I haven’t figured this out yet…)

Since we have not yet implemented transaction related features, we only useblockBarrel. In addition, as mentioned above, we will not store the data of each block in a separate file, but store it in a file. Therefore, we should not store data related to file encoding. In this way, the key value pairs we use are simplified as follows:

  • 32-byte block-hash -> Block structure (serialized)

    Key value pairs of block data and block hash

  • ‘l’ -> the hash of the last block in a chain

    Key value pairs of the latest block hash

serialize

The key and value of rocksdb can only be stored in the form of byte []. Here we need to use the serialization and deserialization libraryKryoThe code is as follows:

package one.wangwei.blockchain.util;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;

/**
 *Serialization tool class
 *
 * @author wangwei
 * @date 2018/02/07
 */
public class SerializeUtils {

    /**
     *Deserialization
     *
     *The byte array corresponding to the @ param bytes object
     * @return
     */
    public static Object deserialize(byte[] bytes) {
        Input input = new Input(bytes);
        Object obj = new Kryo().readClassAndObject(input);
        input.close();
        return obj;
    }

    /**
     *Serialization
     *
     *@ param object the object to be serialized
     * @return
     */
    public static byte[] serialize(Object object) {
        Output output = new Output(4096, -1);
        new Kryo().writeClassAndObject(output, object);
        byte[] bytes = output.toBytes();
        output.close();
        return bytes;
    }
}

Persistence

As mentioned above, we use it hereRocksDBLet’s first write a related tool classRocksDBUtilsIts main functions are as follows:

  • Putlastblockhash: saves the hash value of the latest block
  • Getlastblockhash: query the hash value of the latest block
  • Putblock: save block
  • Getblock: query block

Note: boltdb supports the bucket feature, but rocksdb does not. We use the unified prefix method to handle it.

RocksDBUtils

package one.wangwei.blockchain.util;

import lombok.Getter;
import one.wangwei.blockchain.block.Block;
import org.rocksdb.Options;
import org.rocksdb.RocksDB;
import org.rocksdb.RocksDBException;

/**
 *Rocksdb tool class
 *
 * @author wangwei
 * @date 2018/02/27
 */
public class RocksDBUtils {

    /**
     *Blockchain data file
     */
    private static final String DB_FILE = "blockchain.db";
    /**
     *Blockbucket prefix
     */
    private static final String BLOCKS_BUCKET_PREFIX = "blocks_";

    private volatile static RocksDBUtils instance;

    public static RocksDBUtils getInstance() {
        if (instance == null) {
            synchronized (RocksDBUtils.class) {
                if (instance == null) {
                    instance = new RocksDBUtils();
                }
            }
        }
        return instance;
    }

    @Getter
    private RocksDB rocksDB;

    private RocksDBUtils() {
        initRocksDB();
    }

    /**
     *Initializing rocksdb
     */
    private void initRocksDB() {
        try {
            rocksDB = RocksDB.open(new Options().setCreateIfMissing(true), DB_FILE);
        } catch (RocksDBException e) {
            e.printStackTrace();
        }
    }

    /**
     *Save the hash value of the latest block
     *
     * @param tipBlockHash
     */
    public void putLastBlockHash(String tipBlockHash) throws Exception {
        rocksDB.put(SerializeUtils.serialize(BLOCKS_BUCKET_PREFIX + "l"), SerializeUtils.serialize(tipBlockHash));
    }

    /**
     *Query the hash value of the latest block
     *
     * @return
     */
    public String getLastBlockHash() throws Exception {
        byte[] lastBlockHashBytes = rocksDB.get(SerializeUtils.serialize(BLOCKS_BUCKET_PREFIX + "l"));
        if (lastBlockHashBytes != null) {
            return (String) SerializeUtils.deserialize(lastBlockHashBytes);
        }
        return "";
    }

    /**
     *Save block
     *
     * @param block
     */
    public void putBlock(Block block) throws Exception {
        byte[] key = SerializeUtils.serialize(BLOCKS_BUCKET_PREFIX + block.getHash());
        rocksDB.put(key, SerializeUtils.serialize(block));
    }

    /**
     *Query block
     *
     * @param blockHash
     * @return
     */
    public Block getBlock(String blockHash) throws Exception {
        byte[] key = SerializeUtils.serialize(BLOCKS_BUCKET_PREFIX + blockHash);
        return (Block) SerializeUtils.deserialize(rocksDB.get(key));
    }

}

Create blockchain

Now let’s optimizeBlockchain.newBlockchainThe code logic of the interface is changed to the following logic:

Building blockchain based on Java language (3) -- persistence & command line

The code is as follows:

/**
  *< p > create blockchain</p>
  *
  * @return
  */
public static Blockchain newBlockchain() throws Exception {
    String lastBlockHash = RocksDBUtils.getInstance().getLastBlockHash();
    if (StringUtils.isBlank(lastBlockHash)) {
        Block genesisBlock = Block.newGenesisBlock();
        lastBlockHash = genesisBlock.getHash();
        RocksDBUtils.getInstance().putBlock(genesisBlock);
        RocksDBUtils.getInstance().putLastBlockHash(lastBlockHash);
     }
     return new Blockchain(lastBlockHash);
}

modifyBlockchainOnly the hash value of the latest blockchain is recorded

public class Blockchain {
    
    @Getter
    private String lastBlockHash;

    private Blockchain(String lastBlockHash) {
        this.lastBlockHash = lastBlockHash;
    }
}

After each mining, we also need to save the latest block information and update the latest blockchain hash value:

/**
 *< p > Add block</p>
 *
 * @param data
 */
public void addBlock(String data) throws Exception {
   String lastBlockHash = RocksDBUtils.getInstance().getLastBlockHash();
   if (StringUtils.isBlank(lastBlockHash)) {
       throw new Exception("Fail to add block into blockchain ! ");
   }
   this.addBlock(Block.newBlock(lastBlockHash, data));
}

/**
 *< p > Add block</p>
 *
 * @param block
 */
public void addBlock(Block block) throws Exception {
    RocksDBUtils.getInstance().putLastBlockHash(block.getHash());
    RocksDBUtils.getInstance().putBlock(block);
    this.lastBlockHash = block.getHash();
}

At this point, the function of the storage part is completed. We still lack a function:

Retrieve blockchain

Now, all of our blocks are saved to the database, so we can reopen the existing blockchain and add new blocks to it. But it also makes it impossible for us to print the information of all the blocks in the blockchain, because we don’t store the blocks in the array. Let’s fix this flaw!

We create an inner class in blockchainBlockchainIteratorAs an iterator of the blockchain, it iteratively outputs block information through the hash connection before the block. The code is as follows:

public class Blockchain {
 
    ....
    
    /**
     *Blockchain iterator
     */
    public class BlockchainIterator {

        private String currentBlockHash;

        public BlockchainIterator(String currentBlockHash) {
            this.currentBlockHash = currentBlockHash;
        }

        /**
         *Is there a next block
         *
         * @return
         */
        public boolean hashNext() throws Exception {
            if (StringUtils.isBlank(currentBlockHash)) {
                return false;
            }
            Block lastBlock = RocksDBUtils.getInstance().getBlock(currentBlockHash);
            if (lastBlock == null) {
                return false;
            }
            //Direct release of Chuangshi block
            if (lastBlock.getPrevBlockHash().length() == 0) {
                return true;
            }
            return RocksDBUtils.getInstance().getBlock(lastBlock.getPrevBlockHash()) != null;
        }

        
        /**
         *Return block
         *
         * @return
         */
        public Block next() throws Exception {
            Block currentBlock = RocksDBUtils.getInstance().getBlock(currentBlockHash);
            if (currentBlock != null) {
                this.currentBlockHash = currentBlock.getPrevBlockHash();
                return currentBlock;
            }
            return null;
        }
    }   
    
    ....    
}

test

/**
 *Testing
 *
 * @author wangwei
 * @date 2018/02/05
 */
public class BlockchainTest {

    public static void main(String[] args) {
        try {
            Blockchain blockchain = Blockchain.newBlockchain();

            blockchain.addBlock("Send 1.0 BTC to wangwei");
            blockchain.addBlock("Send 2.5 more BTC to wangwei");
            blockchain.addBlock("Send 3.5 more BTC to wangwei");

            for (Blockchain.BlockchainIterator iterator = blockchain.getBlockchainIterator(); iterator.hashNext(); ) {
                Block block = iterator.next();

                if (block != null) {
                    boolean validate = ProofOfWork.newProofOfWork(block).validate();
                    System.out.println(block.toString() + ", validate = " + validate);
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}


/*Output*/

Block{hash='0000012f87a0510dd0ee7048a6bd52db3002bae7d661126dc28287bd6c23189a', prevBlockHash='0000024b2c23c4fb06c2e2c1349275d415efe17a51db24cd4883da0067300ddf', data='Send 3.5 more BTC to wangwei', timeStamp=1519724875, nonce=369110}, validate = true
Block{hash='0000024b2c23c4fb06c2e2c1349275d415efe17a51db24cd4883da0067300ddf', prevBlockHash='00000b14fefb51ba2a7428549d469bcf3efae338315e7289d3e6dc4caf589d79', data='Send 2.5 more BTC to wangwei', timeStamp=1519724872, nonce=896348}, validate = true
Block{hash='00000b14fefb51ba2a7428549d469bcf3efae338315e7289d3e6dc4caf589d79', prevBlockHash='0000099ced1b02f40c750c5468bb8c4fd800ec9f46fea5d8b033e5d054f0f703', data='Send 1.0 BTC to wangwei', timeStamp=1519724869, nonce=673955}, validate = true
Block{hash='0000099ced1b02f40c750c5468bb8c4fd800ec9f46fea5d8b033e5d054f0f703', prevBlockHash='', data='Genesis Block', timeStamp=1519724866, nonce=840247}, validate = true

Command line interface

CLIPart of the content, here do not do a detailed introduction, specific can see the GitHub source link at the end of the article. The steps are as follows:

to configure

add to pom.xml to configure

<project>
   
    ...
    
    <dependency>
        <groupId>commons-cli</groupId>
        <artifactId>commons-cli</artifactId>
        <version>1.4</version>
    </dependency>
    
    ...
    
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
            <archive>
                <manifest>
                    <addClasspath>true</addClasspath>
                    <classpathPrefix>lib/</classpathPrefix>
                    <mainClass>one.wangwei.blockchain.cli.Main</mainClass>
                </manifest>
            </archive>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
        <executions>
            <execution>
                <id>make-assembly</id>
                <!-- this is used for inheritance merges -->
                <phase>package</phase>
                <! -- specifies to perform the jar package merge operation at the packaging node -- >
                <goals>
                    <goal>single</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    
    ...
   
</project>
Project engineering package
$ mvn clean && mvn package
Execute the order
#Print help information
$ java -jar blockchain-java-jar-with-dependencies.jar -h 

#Add block
$ java -jar blockchain-java-jar-with-dependencies.jar -add "Send 1.5 BTC to wangwei"
$ java -jar blockchain-java-jar-with-dependencies.jar -add "Send 2.5 BTC to wangwei"
$ java -jar blockchain-java-jar-with-dependencies.jar -add "Send 3.5 BTC to wangwei"

#Print blockchain
$ java -jar blockchain-java-jar-with-dependencies.jar -print

summary

In this article, we have implemented the storage function of blockchain. Next, we will implement the functions of address, transaction and wallet.

data

Building blockchain based on Java language (3) -- persistence & command line